In our organization, we’ve undertaken the task of upgrading both SOA and OSB from version 12.2.1.3 to 12.2.1.4. This involves an out-of-place migration approach, wherein we create a new environment and switch traffic during the cutover. As you know, this process typically requires several days we opt for manual provisioing. A very significant amount of automation work was previously accomplished for 12.2.1.3 by our predecessors. Despite these efforts, the process still used to take lot of time due to following factors: Bugs in existing automation. For example, SOA composite deployments is not ..read more

We have a 7 node test rabbitmq cluster. One of the rabbitmq users reported that they were getting “javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature” error while connecting to Rabbitmq. We checked logs and also checked application up time. Rabbitmq servers hadn’t been restarted for the past 25 days. Also, we didn’t observe any errors in logs related to vhost used by client. We also checked if the certificate is expired and it was not. We got reasonably confident that it was not our issue and asked to check from client side. As day progressed, we got complaints from ..read more

In the previous blog post, we discussed about how to monitor weblogic based applications including SOA and OSB using Vmware Wavefront. In this blog post, let us explore open source alternative with Prometheus and Grafana. We will use Oracle Weblogic Monitoring Exporter in place of jolokia agent to export metric to Prometheus and Grafana for visualizing metrics. High level steps are: 1. Download and install Weblogic Monitoring Exporter. 2. Install Prometheus & Grafana 3. Configure Prometheus to scrape metrics from wls-exporter 4. Configure Grafana dashboards using Prometheus datasource. Bel ..read more

Traditionally sysadmins used bash, WLST/Jython scripts to monitor weblogic based applications including Oracle SOA Suite and Oracle Service Bus. There are few disadvantages with this approach: Need to maintain multiple scripts to monitor a single domain. Sysadmins needs to be proficient in 1 or 2 scripting language. Developing script may take few hours to few days. Will not have access to historical monitoring data for doing trend analysis. With the advent of modern monitoring tools like Prometheus and Grafana, above challenges are addressed. We can setup beautiful graphs and dashboards quic ..read more

We were trying to integrate Spring Cloud Gateway (SCG) on Kubernetes with HashiCorp. We followed the steps mentioned in vault documentation. We were able to bring up vault and vault injector but SCG pods were stuck in init state. Found following error in SCG application pod vault-agent-init container logs: NAME READY STATUS RESTARTS AGE scg-0 0/2 Init:0/1 0 9h vault-0 1/1 Running 0 10h vault-agent-injector-5c89c7dfc5-n2v6v 1/1 Running 0 20h Cgo: disabled Log Level: info Version: Vault v1.8.4 Version Sha: 925bc650ad1d997e84fbb832f302a6bfe0105bb ..read more

There are couple of ways to monitor Kafka consumer Lag. There is an open source project called Apache Burrow which has lot of features and easy to set up. It filters out lot of false positives too. If there are an limitations of using it in your project, we can write bash scripts using Kafka consumer utility which ships with Kafka. We have written one such script. If you are interested, please check it out here. The post How to monitor Kafka Consumer Lag appeared first on A Systems Admin Blog ..read more

This blog is about connection reset error we received while connecting from Boomi atom to remote sftp. This issue can occur for any client connecting to a remote sftp server. The sftp server we were trying to connect was old sftp server and it has been used in many integrations. We haven’t faced reset issue for other integrations flow except this new one. So, we first suspected that there was something wrong with the process itself. We looked closely at the process. The process has multiple processes – one main process and multiple child processes. Main process checks for availability of files ..read more

This post is more of a personal note than a perfect analysis. I have applied lot of workarounds and still not able to identify exactly what caused issue. Writing this blog so it will act as future reference if similar issue happens and add more details about the issue. We were adding monitoring for Kafka consumer lag using kafka-consumer-groups.sh which is located under Kafka installation bin directory. We were able to get lag in all environment except one and script execution failed with below error ($KAFKA_BIN/kafka-consumer-groups.sh –describe –all-groups –bootstrap-server localhost:9092 ..read more

We use Boomi extensively for integrations. We have been monitoring Boomi based on existence of Boomi process on VM. This is not capturing all scenarios especially when Boomi process is hung or has cluster issues. To cover monitoring comprehensively, we developed new script based on Boomi Official Documentation recommendations. This script is available at this github location. The post Monitoring Boomi Molecules appeared first on A Systems Admin Blog ..read more

We have faced weird issue while starting Weblogic managed server from admin console. We spent good amount of time in analyzing and fixing the issue. Sharing this if it can help others. We made a configuration change in weblogic domain that required to restart all servers including admin server. We did rolling restart of servers. All servers were started except one of the managed servers. While starting the server from console, we got the error: “the Node Manager associated with machine SOAHOST1 is not reachable“ We logged in into manged server VM and noticed that NM was running fine. Somehow ..read more