Progressive Web Apps (PWA) topic is a hot thing in web development these days. Read more about it - Progressive Web Apps. The beauty and power behind PWA - user can install a web app to his mobile device, without going through the app store. This simplifies update process too, when a new version of the app is available, the user will get it straight away, because it is essentially a Web page, wrapped to look like an installed app.
Inspired by this post - A Simple Progressive Web App Tutorial, I decided to add PWA config into Oracle JET app and test how it works (on Android, didn't test on iOS, but there is nothing JET specific, if PWA is supported on iOS, it should work).
Oracle JET PWA sample app is deployed on Heroku (PWA will work only if the app is coming through HTTPS) and available under this URL. The sample app is available on GitHub repo. Node.js wrapper for this sample is available in another GitHub repo, you can use it to deploy on Heroku or another service.
Access JET app URL, the app will be loaded and you should see Android notification in the bottom. Google Chrome mobile browser automatically is recognizing PWA app by manifest and offers to "install" it by adding to the home screen:
Select notification and you will get a confirmation message:
Select "Add" and Web app will be added to the home screen. It will look like a real mobile app for the user. For example, the user could get runtime stats for the app, check storage and data usage metrics:
The app is added to the home screen (look for Oracle JET icon):
Select the app icon and app will be opened. There is no URL address bar in the header and indeed it looks like a mobile app, not a Web page:
The app will be recognized as PWA, if certain config steps were implemented. One of them - you need to add manifest file (add it in Oracle JET in the same folder as index.html) and provide app icons, name, etc.:
The manifest file must be included through a reference in the app entry point - index page (Oracle JET index.html page for example):
In addition to manifest, the app must define a service worker (same as manifest file, you can create this file in the same directory as Oracle JET index.html). PWA doesn't only bring the visual experience of the real app to the Web application. You can define a cache store for the app files, this means next time when offline - app files will load from local cache storage, there will be no need to download them from the internet:
Service worker can be registered from main.js file where Oracle JET context is initialized on the application initial load. Add service worker registration at the bottom of main.js:
The idea of this post was to share a simple example of PWA for Oracle JET. This should help you to get started quickly with PWA support config for Oracle JET app.
I have implemented JET (more about Oracle JET) showcase app with data visualization components usage. This app shows historical weather data in Boston city, the dataset is taken from Kaggle. Switching years makes data visualization to change and show new data - I love how polar chat is updated. Calendar displays temperature for each day during the year using JET picto chart component:
App is deployed on Heroku and available by this URL. Heroku provides $7 per month account with analytics and better resources, but there is a free option too (it comes with sleep after 30 minutes of inactivity) - free option is good for experimentation, as for this case.
Heroku dashboard for the deployed JET app:
Free deployment comes without analytics option:
App comes with two options - Dashboard and Histogram. The dashboard allows switching between years and shows a polar chart along with daily temperature calendar:
The histogram displays the same data in a different view:
This app comes with Web Component implementation, yes Web Components are a standard feature in JET. Toolbar, where you can switch years, is implemented as Web Component:
Web Component is being used in both UIs - dashboard and histogram:
Visualization components are getting data through Knockout.JS observable variables:
Variables are initialized in JS functions:
1. Heroku deployment guide for Node.js 2. Node.js app which is deployed on Heroku - GitHub. JET content is inside the public folder. JET content is copied from JET app web folder, after running ojet build --release 3. Oracle JET app - GitHub
The sample app is available on GitHub repo. It doesn't require DB connection, you can run it straight away in Oracle JDeveloper.
Look into JSF page. I have implemented ADF Faces input component with value change listener. Below this component, there is HTML div with ID ot1. We will assign a text value to this div programmatically from JS function passClientSideValue:
This is how the end result looks like:
In particular, this approach can be useful, when you want to bypass ADF Faces validation lifecycle and display updated value despite current validation errors in the form.
The goal of this post is to show how convnet (CNN — Convolutional Neural Network) works. I will be using classical cat/dog classification example described in François Chollet book — Deep Learning with Python. Source code for this example is available on François Chollet GitHub. I’m using this source code to run my experiment.
Convnet works by abstracting image features from the detail to higher level elements. An analogy can be described with the way how humans think. Each of us knows how airplane looks, but most likely when thinking about airplane we are not thinking about every little bit of airplane structure. In a similar way, convnet learns to recognize higher level elements in the image and this helps to classify new images when they look similar to the ones used for the training.
Image classification model should be trained using this notebook (you will find a description there from where to download image dataset with cats and dogs images). Model is being used and classification prediction is invoked in this notebook. For the convenience, I uploaded my own notebooks (based on the code from Deep Learning with Python book) to GitHub.
Latest VBCS release brings an option to export VBCS application and run on your own server (or different cloud provider). This is a truly strong step forward for VCBS. Read more about it in Shay Shmeltzer blog post. If you decide to keep running VBCS app within VBCS itself, then you get additional functionality of VBCS Business Services, Oracle Cloud security, etc. out of the box. If you export VBCS application and run on your own environment, these features are not included, but then you don't need to pay for VBCS Cloud runtime when hosting the app. It is great to have alternatives and depending on the customer either one or another of the use cases would work.
One of the use cases - customer even don't need to have its own VBCS instance. We could develop Oracle JET app in our VBCS instance, export and deploy it in the customer environment. Later we could provide support for version upgrade.
I must say it is simple to export VBCS app, no hassle at all. Make sure VBCS app you are exporting is set with anonymous access (this will disable Oracle Cloud security model). You will need to implement security and backend secure calls yourself:
Next go to REST service control and specify Bypass Proxy option (this will enable direct REST service call from VBCS app, bypassing Oracle Cloud proxy service). Important: to work with Bypass Proxy option, REST service must be invoked through HTTPS:
Nothing else on VBCS side. Next need to push application code to Oracle Developer Cloud Service Git repository and build artifact which can be exported. I suggest reading Shay Shmeltzer blog post about how to proceed with VBCS and Oracle Developer Cloud Service setup.
In VBCS do push to Git for the selected app:
If it is the first time with Oracle Developer Cloud Service, you will need to set up (refer to Shay post mentioned above) a build job. Create build job configuration, point to Git repo:
Provide a set of parameters for the build job:
Add Unix Shell script to the build job. This script will execute Node.js NPM command to run vb-build job to construct artifact which can be exported and deployed in your own environment. It is important to make sure that property values used in the script match property values defined in the build job earlier. To execute npm command, make sure to use Oracle Developer Cloud Service machine with Node.js support:
Run the job, once it completes and if there are no errors, go to job artifacts and download optimized.zip - this is the archive with VBCS application you can deploy:
Important: when exported VCBS application is accessed, it loads a bunch of scripts and executes HTTPS requests. There is one request which slows down VBCS application initial loading - call to _currentuser. It is trying to execute the _currentuser request on VBCS instance, but if the instance is down - it will wait until a timeout and only then will proceed with application loading. To fix that, search for _currentuser URL in the exported code and change URL to some dummy value, so that this request will fail immediately and will not keep VBCS application from continue loading.
Is not that complex to build your own chatbot (or assistant, this word is a new trendy term for chatbot) as you may think. Various chatbot platforms are using classification models to recognize user intent. While obviously, you get a strong heads-up when building a chatbot on top of the existing platform, it never hurts to study the background concepts and try to build it yourself. Why not use a similar model yourself. Chatbot implementation main challenges are:
Classify user input to recognize intent (this can be solved with Machine Learning, I’m using Keras with TensorFlow backend)
Keep context. This part is programming and there is nothing much ML related here. I’m using Node.js backend logic to track conversation context (while in context, typically we don’t require a classification for user intents — user input is treated as answers to chatbot questions)
Complete source code for this article with readme instructions is available on my GitHub repo (open source).
This is the list of Python libraries which are used in the implementation. Keras deep learning library is used to build a classification model. Keras runs training on top of TensorFlow backend. Lancaster stemming library is used to collapse distinct word forms:
Chatbot intents and patterns to learn are defined in a plain JSON file. There is no need to have a huge vocabulary. Our goal is to build a chatbot for a specific domain. Classification model can be created for small vocabulary too, it will be able to recognize a set of patterns provided for the training:
Before we could start with classification model training, we need to build vocabulary first. Patterns are processed to build a vocabulary. Each word is stemmed to produce generic root, this would help to cover more combinations of user input:
This is the output of vocabulary creation. There are 9 intents (classes) and 82 vocabulary words:
Training would not run based on the vocabulary of words, words are meaningless for the machine. We need to translate words into bags of words with arrays containing 0/1. Array length will be equal to vocabulary size and 1 will be set when a word from the current pattern is located in the given position:
Training data — X (pattern converted into array [0,1,0,1…, 0]), Y (intents converted into array [1, 0, 0, 0,…,0], there will be single 1 for intents array). Model is built with Keras, based on three layers. According to my experiments, three layers provide good results (but it all depends on training data). Classification output will be multiclass array, which would help to identify encoded intent. Using softmax activation to produce multiclass classification output (result returns an array of 0/1: [1,0,0,…,0] — this set identifies encoded intent):
Fit the model — execute training and construct classification model. I’m executing training in 200 iterations, with batch size = 5:
Model is built. Now we can define two helper functions. Function bow helps to translate user sentence into a bag of words with array 0/1:
Check this example — translating the sentence into a bag of words:
When the function finds a word from the sentence in chatbot vocabulary, it sets 1 into the corresponding position in the array. This array will be sent to be classified by the model to identify to what intent it belongs:
It is a good practice to save the trained model into a pickle file to be able to reuse it to publish through Flask REST API:
Before publishing model through Flask REST API, is always good to run an extra test. Use model.predict function to classify user input and based on calculated probability return intent (multiple intents can be returned):
Example to classify sentence:
The intent is calculated correctly:
To publish the same function through REST endpoint, we can wrap it into Flask API:
Flask is fun and easy to setup, as it says on Flask website. And that's true. This microframework for Python offers a powerful way of annotating Python function with REST endpoint. I’m using Flask to publish ML model API to be accessible by the 3rd party business applications.
This example is based on XGBoost.
For better code maintenance, I would recommend using a separate Jupyter notebook where ML model API will be published. Import Flask module along with Flask CORS:
Model is trained on Pima Indians Diabetes Database. CSV data can be downloaded from here. To construct Pandas data frame variable as input for model predict function, we need to define an array of dataset columns:
Previously trained and saved model is loaded using Pickle:
It is always a good practice to do a test run and check if the model performs well. Construct data frame with an array of column names and an array of data (using new data, the one which is not present in train or test datasets). Calling two functions — model.predict and model.predict_proba. Often I prefer model.predict_proba, it returns probability which describes how likely will be 0/1, this helps to interpret the result based on a certain range (0.25 to 0.75 for example). Pandas data frame is constructed with sample payload and then the model prediction is executed:
Flask API. Make sure you enable CORS, otherwise API call will not work from another host. Write annotation before the function you want to expose through REST API. Provide an endpoint name and supported REST methods (POST in this example). Payload data is retrieved from the request, Pandas data frame is constructed and model predict_proba function is executed:
Response JSON string is constructed and returned as a function result. I’m running Flask in Docker container, that's why using 0.0.0.0 as the host on which it runs. Port 5000 is mapped as external port and this allows calls from the outside.
While it works to start Flask interface directly in Jupyter notebook, I would recommend to convert it to Python script and run from command line as a service. Use Jupyter nbconvert command to convert to Python script:
jupyter nbconvert — to python diabetes_redsamurai_endpoint_db.ipynb
Python script with Flask endpoint can be started as the background process with PM2 process manager. This allows to run endpoint as a service and start other processes on different ports. PM2 start command:
pm2 start diabetes_redsamurai_endpoint_db.py
pm2 monit helps to display info about running processes:
ML model classification REST API call from Postman through endpoint served by Flask:
- GitHub repo with source code - Previous post about XGBoost model training
There is always a bit of luck involved when selecting parameters for Machine Learning model training. Lately, I work with gradient boosted trees and XGBoost in particular. We are using XGBoost in the enterprise to automate repetitive human tasks. While training ML models with XGBoost, I created a pattern to choose parameters, which helps me to build new models quicker. I will share it in this post, hopefully you will find it useful too.
This is the Python code which runs XGBoost training step and builds a model. Training is executed by passing pairs of train/test data, this helps to evaluate training quality ad-hoc during model construction:
Key parameters in XGBoost (the ones which would affect model quality greatly), assuming you already selected max_depth (more complex classification task, deeper the tree), subsample (equal to evaluation data percentage), objective (classification algorithm):
n_estimators — the number of runs XGBoost will try to learn
learning_rate — learning speed
early_stopping_rounds — overfitting prevention, stop early if no improvement in learning
When model.fit is executed with verbose=True, you will see each training run evaluation quality printed out. At the end of the log, you should see which iteration was selected as the best one. It might be the number of training rounds is not enough to detect the best iteration, then XGBoost will select the last iteration to build the model.
With matpotlib library we can plot training results for each run (from XGBoost output). This helps to understand if iteration which was chosen to build the model was the best one possible. Here we are using sklearn library to evaluate model accuracy and then plotting training results with matpotlib:
Let’s describe my approach to select parameters (n_estimators, learning_rate, early_stopping_rounds) for XGBoost training.
Step 1. Start with what you feel works best based on your experience or what makes sense
n_estimators = 300
learning_rate = 0.01
early_stopping_rounds = 10
Stop iteration = 237
Accuracy = 78.35%
With the first attempt, we already get good results for Pima Indians Diabetes dataset. Training was stopped at iteration 237. Classification error plot shows a lower error rate around iteration 237. This means learning rate 0.01 is suitable for this dataset and early stopping of 10 iterations (if the result doesn’t improve in the next 10 iterations) works.
Step 2. Experiment with learning rate, try to set a smaller learning rate parameter and increase number of learning iterations
n_estimators = 500
learning_rate = 0.001
early_stopping_rounds = 10
Stop iteration = didn’t stop, spent all 500 iterations
Accuracy = 77.56%
Smaller learning rate wasn’t working for this dataset. Classification error almost doesn’t change and XGBoost log loss doesn’t stabilize even with 500 iterations.
Step 3. Try to increase the learning rate.
n_estimators = 300
learning_rate = 0.1
early_stopping_rounds = 10
Stop iteration = 27
Accuracy = 76.77%
With increased learning rate, the algorithm learns quicker, it stops already at iteration Nr. 27. XGBoost log loss error is stabilizing, but the overall classification accuracy is not ideal.
Step 4. Select optimal learning rate from the first step and increase early stopping (to give the algorithm more chances to find a better result).
n_estimators = 300
learning_rate = 0.01
early_stopping_rounds = 15
Stop iteration = 265
Accuracy = 78.74%
A slightly better result is produced with 78.74% accuracy — this is visible in the classification error plot.
The process to prepare data for Machine Learning model training to me looks somewhat similar to the process of preparing food ingredients to cook dinner. You know in both cases it takes time, but then you are rewarded with tasty dinner or a great ML model.
I will not be diving here into data science subject and discussing how to structure and transform data. It all depends on the use case and there are so many ways to reformat data to get the most out of it. I will rather focus on simple, but a practical example — how to split data into training and test datasets with Python.
Oracle JET table comes with template slot option. This is helpful to build generic functionality to render custom cell within the table.
In this example, custom cells are used to render dates, amount and risk gauge:
While implementing Oracle JET table it is a best practice to read table column structure from a variable, not to define the entire structure in HTML itself. Property columns refer to the variable. Template called cellTemplate is a default template to render cell content:
Table column structure is defined in JS. To apply specific cell template, it is specified in column definition:
Table data is static in this example and coming through JSON array based on JET Array Data Provider: