Loading...

Follow Paul Shapley's | Open Source Mapping & GIS on Feedspot

Continue with Google
Continue with Facebook
or

Valid
For those potential GI users who are too busy to learn GIS skills but need to know where those boundaries are, this could be the answer for you. Through a simple leaflet map we can connect directly into our spatial data in PostgreSQL/PostGIS data with 'R' and 'Shiny'. For more information click the link.

https://shiny.rstudio.com/

In the example below we are going to install 'RStudio-Server' on an Ubuntu 18.04 LTS server in order  to run an 'R' script although I could just as easily have created this script using any other IDE or text editor if preferred. 'RStudio' is a complete environment for prototyping web apps (known as 'Shiny' apps) which allow the user to interact with the data imported from various sources for example to look into changes over a ten year timescale interactively from an excel spreadsheet.

'RStudio-Server' is a free and open-source integrated development environment (IDE) for R, a programming language for primarily statistical computing, graphics and mapping analysis (it is a full gis with input, analysis and output 'packages' utilising many of our familiar OSGEO libraries used in QGIS, SAGA, GRASS, i.e. GDAL, PROJ4, GEOS etc but more focused on statistical and web output).

The 'RStudio' Interface, 'script' top left, 'console' lower left and 'output' lower right.

There are minor similarities to the 'qgis2web' plugin (for QGIS) with some 'Power BI' style tools thrown in the mix with more functionality to 'fine-tune' your data output whether an excel spreadsheet or a database table or from any other source. There are other important differences between 'R' compared to standard HTML/CSS/PHP/Javascript web frameworks in that there is only one script containing all the above tools including the UI and Server functionality.

I admit here that I am not a developer or coder but it is really easy to pick up once you get started. A great tool for fast prototyping. There is also an interesting discussion around 'R Shiny' v 'Power BI' if anyone is interested.

https://www.musgraveanalytics.com/blog/2018/1/19/r-shiny-vs-power-bi

I'd been looking for a tool to get map data stored in a Postgres/PostGIS tables on the web quickly for non traditional GIS users to be able to view the data and for me to be able to modify the code changing features on the fly without having to restart the server.

Let us begin and get 'RStudio-Server' installed.
Some text conventions before we start:-

----------------------------------------------------------------------------------

$ red instructions = manually enter terminal command.
Red italics with underline = add your own info/config here.
# Instruction = non-parsing instructions or comments preceding the command.

----------------------------------------------------------------------------------

GENERAL ACTIONS REQUIRED:-

#If you don't have 'R' installed already (on UBUNTU 18.04 SERVER in this case).

$ sudo apt-get install r-base r-base-dev 

#(R 3.6) (if package incompatibilities arise or above packages won't install then delete:- /usr/local/lib/R) then (remove --purge, autoremove) and install again as above.

#install dependencies

$ sudo apt-get install gdebi-core

#download the shiny server packages

wget http://download3.rstudio.org/ubuntu-18.04/x86_64/shiny-server-1.5.12.931-amd64.deb

#install shiny-server-1.5.12.931-amd64.deb

$ sudo gdebi shiny-server-1.5.12.931-amd64.deb

#To access rstudio-server v1.2.1.1335 thereafter

http://XXX.XX.X.XX:8787 user:your_user pass:your_password (DEFAULTS TO PORT '8787' ON YOUR SERVER)

#main project location

/opt/shiny-server/samples/sample-apps/your_shiny_app_directory_here/app.R

--------------------------------------------------------------------------

#Install the 'cran' 'R' packages (in a 'Global' library so all users can access them) used in your app starting with 'devtools'

$ sudo su - your_user

$ sudo apt-get -y install libcurl4-gnutls-dev libxml2-dev libssl-dev

$ sudo su - -c "R -e \"install.packages('devtools', repos='http://cran.rstudio.com/')\""
$ sudo su - -c "R -e \"install.packages('shiny', repos='http://cran.rstudio.com/')\""your_user
$ sudo su - -c "R -e \"install.packages('shinythemes', repos='http://cran.rstudio.com/')\""
$ sudo su - -c "R -e \"install.packages('rsconnect', repos='http://cran.rstudio.com/')\""
$ sudo su - -c "R -e \"install.packages('RPostgreSQL', repos='http://cran.rstudio.com/')\""
$ sudo su - -c "R -e \"install.packages('rpostgis', repos='http://cran.rstudio.com/')\""
$ sudo su - -c "R -e \"install.packages('sp', repos='http://cran.rstudio.com/')\""
$ sudo su - -c "R -e \"install.packages('rgdal', repos='http://cran.rstudio.com/')\"" (**requires 'libgdal-dev' on sudo apt install** ln -s  )
$ sudo su - -c "R -e \"install.packages('leaflet', repos='http://cran.rstudio.com/')\""
$ sudo su - -c "R -e \"install.packages('leaflet.extras', repos='http://cran.rstudio.com/')\""

Once this has been done we can remove the 'install.packages("")' parts of the 'app.R' script

/opt/shiny-server/samples/sample-apps/your_shiny_app_directory_here/app.R

-------------------------------------------------------------------------

#Upload folders/files with (use winSCP on Windows Servers) the directory (your_shiny_app_directory_here) to /srv/shiny-server/ then move it to the main server directory:-

$ sudo mv /srv/shiny-server/your_shiny_app_directory_here/ /opt/shiny-server/

#Then 'symlink' back to /srv/shiny-server along with 'sample-apps' folder and 'index.html' (used for testing).

$ sudo ln -s /opt/shiny-server/your_shiny_app_directory_here/srv/shiny-server/your_shiny_app_directory_here
$ sudo ln -s /opt/shiny-server/samples/welcome.html /srv/shiny-server/index.html
$ sudo ln -s /opt/shiny-server/samples/sample-apps /srv/shiny-server/sample-apps


-------------------------------------------------------------------------

/etc/shiny-server/shiny-server.conf

A default 'shiny-server.conf' configuration file looks like this:-

-------------------------------------------------------------------------

preserve_logs true

# Instruct Shiny Server to run applications as the user "shiny"

run_as your_user;
# Define a server that listens on port '3838'
server {
  listen 3838;

  # Define a location at the base URL
  location / {

    # Host the directory of Shiny Apps stored in this directory
    site_dir /srv/shiny-server;

    # Log all Shiny output to files in this directory
    log_dir /var/log/shiny-server;

    # When a user visits the base URL rather than a particular application,
    # an index of the applications available in this directory will be shown.
    directory_index on;
  }
}

----------------------------------------------------------------------

DONT FORGET: -

$ sudo systemctl restart shiny-server

LOG LOCATIONS:-

/var/log/shiny-server (shiny-server specific)

leaflet zoom levels:-

5km = Zoom 10
3km = Zoom 11
2km = Zoom 12

1km = Zoom 13

---------------------------------------------------------------------
The 'R' Script (download here)
Red italics with underline = add your own info here

Before copying this script into the script editor if you are using any images within the page please ensure they are situated within the 'www' directory.

# To create and test this script in RStudio you'll need to install the packages first if you've installed them 'Globally' on your server then remove them here in the script

install.packages("shiny")
install.packages("shinythemes")
install.packages("rsconnect")
install.packages("RPostgreSQL")
install.packages("rpostgis")
install.packages("sp")
install.packages("rgdal")
install.packages("leaflet")
install.packages("leaflet.extras")

# Install the libraries

library(shiny)
library(shinythemes)
library(rsconnect)
library(RPostgreSQL)
library(rpostgis)
library(sp)
library(rgdal)
library(leaflet)
library(leaflet.extras)

# Specify Driver

pg <- dbDriver("PostgreSQL")

# Connection Details

con <- dbConnect(pg, user ='your_user', password ="your_password_here", host = "your_host_here" , port = 5432, dbname="your_db_here")

# List all Tables in database (to test the connection is working)

dbListTables(con)

# Read the PostGIS table "layer_1" into a 'data.frame' for the dynamic labels

dbReadTable(con, "layer_1", schema = "layer_1_schema_here") %>% head(3)

# list the Fields in the table 

dbListFields(con, "layer_1", schema = "layer_1_schema_here")

# Query the tables for each layer you require

query <- "SELECT * FROM layer_1"
layer_1 <- pgGetGeom(con, c("layer_1_schema_here","layer_1"), geom = "wkb_geometry", gid = "ogc_fid", other.cols = TRUE)

query <- "SELECT * FROM layer_2"
layer_2 <- pgGetGeom(con, c("layer_2_schema_here","layer_2"), geom = "wkb_geometry", gid = "ogc_fid", other.cols = TRUE)

# Plot the layer in Viewer

plot(layer_1,axes=TRUE)
plot(layer_2,axes=TRUE)

# Change the projection from OSGB36 (epsg:27700) to WGS 84 (epsg:4326)

layer_1 <- spTransform(layer_1, CRS("+init=epsg:4326"))
layer_2 <- spTransform(layer_2, CRS("+init=epsg:4326"))

# UI Function

ui <- fluidPage(theme = shinytheme("cerulean"),
                tags$head(HTML("<title>myTitle</title> <link type='image/png' href='myImage.png'>")),
                titlePanel(title=div(img(src="myImage.png", height = 100, width = 100, align = "left","myTitle"))),
                h4("My h4 text here."),
                p("My paragraph text here"),
                leafletOutput("layer_1", height = 500))

# Server Function

server <- function(input, output, session)
  
output$layer_1 <- renderLeaflet ({
    map <- leaflet(layer_1) %>%
      flyTo(-0.0000, 00.0000, zoom = 10) %>%
      addTiles(group="OpenStreetMap.Mapnik") %>% 
      addProviderTiles("Esri.WorldImagery", group = "Satellite") %>%
      addProviderTiles("OpenStreetMap.HOT", group = "OpenStreetMap.HOT") %>%
      addProviderTiles("Stamen.TerrainBackground", group = "Stamen.TerrainBackground", options = providerTileOptions(minzoom = 1, maxzoom = 13)) %>%
      addMiniMap(tiles = "OpenStreetMap.HOT", toggleDisplay = TRUE) %>%
      addSearchOSM() %>%
      addEasyButton(easyButton(icon="fa-globe", title="Zoom to Level 1", onClick=JS("function(btn, map){ map.setZoom(1); }"))) %>%
      addEasyButton(easyButton(icon="fa-crosshairs", title="Locate Me", onClick=JS("function(btn, map){ map.locate({setView: true}); }"))) %>%
      addPolygons(data=layer_1
                  fillColor = "#e41a1c", 
                  fillOpacity = 0.4, 
                  stroke = "#690E0E",
                  dashArray = "4",
                  weight = 0.7,
                  label = ~name_of_your_table_attribute_column_to_label_here,
                  labelOptions = labelOptions(style = list("font-weight" = "normal", padding = "3px 8px"), textsize = "13px", direction = "auto"),
                  group = "layer_1",
                  highlight = highlightOptions(color = "black", weight = 4, bringToFront = TRUE)) %>%
      addPolygons(data=layer_2
                  fillColor = "#377eb8", 
                  fillOpacity = 0.4, 
                  stroke = "#20445F",
                  dashArray = "4",
                  weight = 0.7,
                  label = ~yourLabel,
                  labelOptions = labelOptions(style = list("font-weight" = "normal", padding = "3px 8px"), textsize = "13px", direction = "auto"),
                  group = "layer_2",
                  highlight = highlightOptions(color = "black", weight = 4, bringToFront = TRUE)) %>%
      addLegend("bottomleft", title = "Map Legend", colors = c("#e41a1c","#377eb8"), labels = c("layer_1","layer_2")) %>%
      addFullscreenControl() %>%
      addMeasure() %>%
      addScaleBar(position = c("bottomleft")) %>%
      addLayersControl(baseGroups = c("OpenStreetMap.Mapnik","Satellite","OpenStreetMap.HOT","Stamen.TerrainBackground"),
                       overlayGroups = c("layer_1","layer_2"),
                       options = layersControlOptions(collapsed = TRUE))
    map
  })

# ShinyApp Function

shinyApp(ui = ui, server = server)




  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
I never thought I would need a 'to-do' list or a 'note taking' application before. I turn up at meetings with my trusty black pocket book and a pen. How 'old school' I hear you say but I am always late to the party. I signed up to Evernote some years ago but that is as far as I got. Now I cannot stop adding stuff to remember in 'Joplin'. So what is so different about this syncing 'to-do' app (compared to the Evernote free version)?

Evernote has no 'markdown' support.
Evernote has no 'end to end' encryption.
Evernote has a two device limit.
Evernote cannot work offline.
Evernote isn't open source.

The name 'Joplin' comes from composer Scott Joplin

"Joplin is a free, open source note taking and to-do application, which can handle a large number of notes organised into notebooks. The notes are searchable, can be copied, tagged and modified either from the applications directly or from your own text editor. The notes are in Markdown format.
Notes exported from Evernote via .enex files can be imported into Joplin, including the formatted content (which is converted to Markdown), resources (images, attachments, etc.) and complete metadata (geolocation, updated time, created time, etc.). Plain Markdown files can also be imported.
The notes can be synchronised with various cloud services including Nextcloud, Dropbox, OneDrive, WebDAV or the file system (for example with a network directory). When synchronising the notes, notebooks, tags and other metadata are saved to plain text files which can be easily inspected, backed up and moved around.
The application is available for Windows, Linux, macOS, Android and iOS. A Web Clipper, to save web pages and screenshots from your browser, is also available for Firefox and Chrome". Source:-
https://joplin.cozic.net/

Installation on Ubuntu (copy and paste) in terminal and press 'return'

wget -O - https://raw.githubusercontent.com/laurent22/joplin/master/Joplin_install_and_update.sh | bash

Thanks to my colleague Phil Barlow for introducing me to it.
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
I'm extremely proud that I can play a very small part in this amazing project to get Ubuntu Touch on Mobile devices.

https://ubuntu-touch.io/get-ut
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Install Ubuntu 18.04 LTS (Bionic Beaver)
1. check your Ubuntu version:-

lsb_release -a

Install Postgresql
2. On the server or your PC (Terminal):-

sudo apt-get update
sudo apt-get install -y postgresql postgresql-contrib

Create a database and a user for access
Replace DATABASE_NAME_HERE and USER_NAME_HERE with the values you want to use.
# this will prompt you for a database password...also note the capital letter 'O' not number '0' (zero) below:-

3. sudo -u postgres createuser -P USER_NAME_HERE
sudo -u postgres createdb -O USER_NAME_HERE DATABASE_NAME_HERE

Test Connection to Postgresql
4. psql -h localhost -U USER_NAME_HERE DATABASE_NAME_HERE

Postgresql will ask you for your password. Then you should see the following:

DATABASE_NAME_HERE=>

To exit type:-

\q

To add PostGIS-2.4 support to the database
5. sudo apt-get install -y postgis postgresql-10-postgis-2.4


sudo -u postgres psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_topology;" DATABASE_NAME_HERE

6. Finally... install PgAdmin3 and set your 'service' connections

 sudo apt-get install pgadmin3

My thanks to Jon Saints for his original post dated from 13th August 2014 (for Ubuntu 14.04)
http://www.saintsjd.com/2014/08/13/howto-install-postgis-on-ubuntu-trusty.html
http://www.github.com/saintsjd
http://facebook.com/saintsjd

Finally...you can check your terminal output with the one below:-

:~$ sudo apt install -y postgresql postgresql-contrib
[sudo] password for xxxx:
Reading package lists... Done
Building dependency tree     
Reading state information... Done
Suggested packages:
  postgresql-doc
The following NEW packages will be installed
  postgresql postgresql-contrib
0 to upgrade, 2 to newly install, 0 to remove and 27 not to upgrade.
Need to get 11.6 kB of archives.
After this operation, 127 kB of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu bionic/main amd64 postgresql all 10+190 [5,784 B]
Get:2 http://gb.archive.ubuntu.com/ubuntu bionic/main amd64 postgresql-contrib all 10+190 [5,796B]
Fetched 11.6 kB in 0s (101 kB/s)             
Selecting previously unselected package postgresql.
(Reading database ... 399415 files and directories currently installed.)
Preparing to unpack .../postgresql_10+190_all.deb ...
Unpacking postgresql (10+190) ...
Selecting previously unselected package postgresql-contrib.
Preparing to unpack .../postgresql-contrib_10+190_all.deb ...
Unpacking postgresql-contrib (10+190) ...
Setting up postgresql-contrib (10+190) ...
Setting up postgresql (10+190) ...
:~$ sudo -u postgres createuser -P postgis
Enter password for new role: xxxx (your password)
Enter it again: xxxx (your password)
:~$ sudo -u postgres createdb -O postgis webgis
createdb: database creation failed: ERROR:  database "webgis" already exists
:~$ sudo -u postgres createdb -O postgis polybase
:~$ psql -h localhost -U postgis polybase
Password for user postgis:
psql (10.6 (Ubuntu 10.6-0ubuntu0.18.04.1), server 9.6.8)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

polybase=>

\q

:~$ sudo apt install -y postgis postgresql-10-postgis-2.4
Reading package lists... Done
Building dependency tree     
Reading state information... Done
postgis is already the newest version (2.4.3+dfsg-4).
The following NEW packages will be installed
  libprotobuf-c1 postgresql-10-postgis-2.4
0 to upgrade, 2 to newly install, 0 to remove and 27 not to upgrade.
Need to get 578 kB of archives.
After this operation, 1,748 kB of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu bionic/universe amd64 libprotobuf-c1 amd64 1.2.1-2[19.0 kB]
Get:2 http://gb.archive.ubuntu.com/ubuntu bionic/universe amd64 postgresql-10-postgis-2.4 amd64 2.4.3+dfsg-4 [559 kB]
Fetched 578 kB in 0s (2,861 kB/s)               
Selecting previously unselected package libprotobuf-c1:amd64.
(Reading database ... 399421 files and directories currently installed.)
Preparing to unpack .../libprotobuf-c1_1.2.1-2_amd64.deb ...
Unpacking libprotobuf-c1:amd64 (1.2.1-2) ...
Selecting previously unselected package postgresql-10-postgis-2.4.
Preparing to unpack .../postgresql-10-postgis-2.4_2.4.3+dfsg-4_amd64.deb ...
Unpacking postgresql-10-postgis-2.4 (2.4.3+dfsg-4) ...
Setting up libprotobuf-c1:amd64 (1.2.1-2) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Setting up postgresql-10-postgis-2.4 (2.4.3+dfsg-4) ...
:~$ sudo -u postgres psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_topology;"polybase
ERROR:  syntax error at or near "polybase"
LINE 1: ...XTENSION postgis; CREATE EXTENSION postgis_topology;polybase
                                                               ^
:~$ sudo apt install pgadmin4
Reading package lists... Done
Building dependency tree     
Reading state information... Done
Package pgadmin4 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'pgadmin4' has no installation candidate

:~$ sudo apt install pgadmin3
Reading package lists... Done
Building dependency tree     
Reading state information... Done
pgadmin3 is already the newest version (1.22.2-4).
0 to upgrade, 0 to newly install, 0 to remove and 27 not to upgrade.
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Selecting and extracting subtle landscape changes to a shapefile using Grass GIS and QGIS - YouTube
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
My previous post didn't mention how to setup the Ordnance Survey layers (or any other raster and vector combination) in a single GeoPackage from virtual rasters, so I thought i would elaborate on the 'gdal_translate' part. It's quite easy...you can simply cut and paste this command in Linux or Windows. Remember to substitute with your own paths with your virtual raster layers.

The advantages of the GeoPackage format in QGIS3 are not only speed of map scrolling and zooming but the ability to store (in a database table) your map 'styles' and zoom levels (for vector layers) within when needing to work offline or with Qfield. Also remember that you do not need to include the .vrt extension.

Before you start install and open QGIS 3.0 and then open the 'Browser' and select 'GeoPackage' then right click 'Create database'.



You will then see the panel below. Enter a database name for your new GeoPackage and leave everything else as 'default', then press 'OK'. You should now see a new empty GeoPackage at the location you specified.




Now open the Terminal/Commandline and type the following one layer/block at a time depending on the number of .vrt layers you have.

:~$ gdal_translate /your/path/to/vrtfile/OS_GB_Overview_19-09-17/OS_GB_Overview_19-09-17 /your/path/to/gpkg/psma.gpkg -of GPKG -co APPEND_SUBDATASET=YES -co RASTER_TABLE=OS_GB_Overview_19-09-17 -co ZOOM_LEVEL_STRATEGY=AUTO

:~$ gdal_translate /your/path/to/vrtfile/OS_MiniScale_19-09-17/OS_MiniScale_19-09-17 /your/path/to/gpkg/psma.gpkg -of GPKG -co APPEND_SUBDATASET=YES -co RASTER_TABLE=OS_MiniScale_19-09-17 -co ZOOM_LEVEL_STRATEGY=AUTO

:~$ gdal_translate /your/path/to/vrtfile/250k_19-09-17/250k_19-09-17 /your/path/to/gpkg/psma.gpkg -of GPKG -co APPEND_SUBDATASET=YES -co RASTER_TABLE=250k_19-09-17 -co ZOOM_LEVEL_STRATEGY=AUTO

:~$ gdal_translate /your/path/to/vrtfile/50k_19-09-17/50k_19-09-17 /your/path/to/gpkg/psma.gpkg -of GPKG -co APPEND_SUBDATASET=YES -co RASTER_TABLE=50k_19-09-17 -co ZOOM_LEVEL_STRATEGY=AUTO

:~$ gdal_translate /your/path/to/vrtfile/25k_19-09-17/25k_19-09-17 /your/path/to/gpkg/psma.gpkg -of GPKG -co APPEND_SUBDATASET=YES -co RASTER_TABLE=25k_19-09-17 -co ZOOM_LEVEL_STRATEGY=AUTO

:~$ gdal_translate /your/path/to/vrtfile/VML_19-09-17/VML_19-09-17 /your/path/to/gpkg/psma.gpkg -of GPKG -co APPEND_SUBDATASET=YES -co RASTER_TABLE=VML_19-09-17 -co ZOOM_LEVEL_STRATEGY=AUTO
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
GeoPackage Ordnance Survey Data for QGIS 3 - YouTube
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
QGIS User 0002 - GeoPackage with Rasters - YouTube

Read for later

Articles marked as Favorite are saved for later viewing.
close
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Separate tags by commas
To access this feature, please upgrade your account.
Start your free month
Free Preview