Loading...

Follow Code-diesel Blog on Feedspot

Continue with Google
Continue with Facebook
or

Valid

A recent debugging session on a web application surfaced a recurrent issue in database design – that of assigning a wrong data type to a database field.

Storing numeric data in a numeric field, when is it not recommended?

Many programmers, when designing a database schema, default to using a numeric data type for a numeric field – like zip codes, telephone numbers, age, price etc. However, not every numeric value needs to be stored as a numeric type. Many times it is prudent to store numeric values as a ‘text’ type.

Take the case of a zip code – it usually gets assigned a numeric type. However, if you check zip codes in the US you can see that many of them start with a zero. A few random examples are given below.

07097	JERSEY CITY
07403	BLOOMINGDALE
00610	ANASCO
01541	PRINCETON

I’ve a database of around 43,000 US zip codes, out of which 3,000 (~ 7%) zip codes start with a zero. Saving this zip codes in a numeric field will truncate the initial zeros, losing information in the process.

This actually happened with a recent client project. I was debugging a legacy web application which was used by ground service teams to assign service personnel for home TV installations. Upon arrival of a new request the technician closest to the client location was assigned the job. This required calculating distances using the client zip code and the technicians location. However the application was sometimes giving wrong distances, giving the location distance as thousands of miles when actually it was just 12 miles. Digging around it was noticed that the corresponding zip codes were truncated and hence the wrong distances.

The data loss probably occurred when importing a ‘zip codes’ CSV into the database. As the database ‘zip codes’ field was set to a numeric type, while importing the initial zeros got truncated.

So when should numeric data be stored in a numeric type field and when in a ‘text’ type. The rule of thumb is that if you are going to do calculations on the numeric data – addition, subtraction, averaging etc. than use a numeric field type to store the data. You do not, or rather cannot, make any meaningful calculations on a zip code or a telephone number, so it is wise to store it as a text ‘char’ type.

Contrast zip codes with a ‘age’ field. One can do various calculations on it – average, min, max, standard deviation etc. So one can, or should, use a numeric field type for ‘age’.

The post Pitfalls of assigning a wrong data type to a database column appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

PHP command-line interface (CLI) includes a nifty option to quickly check for any syntax errors in a source code file. A simple check for a single file is given below. The option flag to check is -l (lowercase ‘L’).

$ php -l example.php

If the file contains no syntax errors, the CLI returns the following message.

No syntax errors detected in example.php

If there is an error in the code the CLI will return the error depending on the context of the error.

Parse error: syntax error, unexpected '$dom' (T_VARIABLE) in example.php on line 6
Errors parsing example.php

We can also test for multiple PHP files in the current directory using a BASH shell command.

$ for i in *.php; do php -l $i; done

This will test all the PHP files in the current directory for any syntax errors and report the same.

On Windows you can use the following in the command windows to check for the same. Assuming that your command window is in the directory where the PHP files are located.

c:\localhost\test\> for %x in (*.php) do @php -l "%x"

Besides returning a message, the shell also returns a exit code of ‘-1’ if the checking fails or a ‘0’ if no errors are found. This can be useful if you want to test it via a shell script or another PHP program.

The post Check PHP syntax from the command line appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

If you have recently moved or are thinking of moving to the nodejs Express framework than you should primarily get acquainted with its routing and middleware concepts, as an Express application is essentially a series of middleware function calls. Middlewares in Express are functions executed after the incoming request from the client, the middleware then processes the request depending on its purpose and then hands over the request to the next middleware in the chain or generates a output if is the last one in the chain, terminating the request-response cycle.

Middleware in a nutshell

Middleware are functions that primarily work with the two main objects in an Express application – the request object (req) and the response object (res). It also has access to the ‘next‘ function which is the function next in the application’s request-response cycle. The next function is a function in the Express router which, when invoked, executes the middleware succeeding the current middleware.

Middleware functions primarily perform the following tasks:

a. Execute code.
b. Make changes to the request and/or the response objects.
c. End the request-response cycle, or
d. Call the next middleware in the stack using next().

If the current middleware function does not end the request-response cycle, it must call next() to pass control to the next middleware function. Otherwise, the request will be left hanging without anything to do further.

An example

Take the following basic Express application which has the singular job of printing ‘Hello World!’ in the browser.

var express = require('express');
var app = express();
 
app.get('/', function (req, res) {
  res.send('Hello World!');
})
 
app.listen(3000);

Now what we want to do is log all urls along with their access times to the console. Rather than adding the code to the ‘get’ route we can create a middleware which allows us to do that without messing around with existing code.

var express = require('express');
var app = express();
 
/* url logger middleware */
var urlLogger = function (req, res, next) {
  console.log(Date(Date.now()) + " : " + req.url);
  next();
}
 
app.use(urlLogger);
 
app.get('/', function (req, res) {
  res.send('Hello World!');
})
 
app.listen(3000);

Now each time we request the web application the url of the request is logged in the console, along with the time as shown below.

Tue Jan 15 2019 14:05:34 GMT+0530 (GMT+05:30) : /welcome
Middleware cautions

Note the next() function call at the end of the above urlLogger middleware. This essentially hands over the request to the next middleware in the chain. If this is accidentally omitted then the application will have nothing to handover the request further to and will keep waiting. In our above example the urlLogger middleware will hand over the request to the next in line – the ‘get’ route. As you can see the ‘get’ route does not have the next() call which basically means this is the last middleware in the chain.

Note: The order of middleware loading is important – middleware functions that are loaded first are also executed first. If urlLogger is loaded after the route to the root path in our example, the request never reaches the ‘get’ route and the app doesn’t print the url details, because the route handler of the root path terminates the request-response cycle.

If we define the middleware later then the middleware will not be executed on the request.

/* url logger middleware will not be executed on the following 'get' request
   as it is defined later.
 */
app.get('/', function (req, res) {
  res.send('Hello World!');
})
 
app.use(urlLogger);

As we have defined urlLogger on global app basis, this middleware will be executed on all app requests or routes. You can use the urlLogger middleware on select routes by removing the following line which defines it on a global basis.

app.use(urlLogger);

And specifying the middleware in the route as required. Lets us say we need to only log requests to the ‘/welcome’ route. The following will accomplish the same.

app.get('/welcome', urlLogger, function (req, res) {
  res.send('Welcome!');
})

Note that we can name the ‘next’ function anything, but keep it consistent throughout your application. I prefer ‘next’ as this is the official convention. Below we have name it ‘done’.

var urlLogger = function (req, res, done) {
  console.log(Date(Date.now()) + " : " + req.url);
  done();
}
In conclusion

There are many third-party middlewares that will help you to build your Express application. A commonly used is ‘body-parser‘ – a middleware that parses incoming request bodies in a middleware before your handlers and which is available under the ‘req.body’ property.

In the next post we will see more on how to define Express middleware for various routes.

The post Middleware in Express applications appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Most terminal applications or tools in NodeJS are single colored in nature. ‘Chalk’ is a nodejs package that allows you to style your application terminal output strings in various colors and styles, making it more readable and colorful.

Installation is via npm.

npm install chalk

Usage is short and simple.

const chalk = require('chalk');
console.log(chalk.blue('Hello world!')); // Displays string in Red

A more detailed example.

const chalk = require('chalk');
 
const log = console.log;
 
// Combine styled and normal strings
log(chalk.blue('Hello') + ' World' + chalk.red('!'));
 
// Compose multiple styles using the chainable API
log(chalk.blue.bgYellow.bold('Hello world!'));
 
// Pass in multiple arguments
log(chalk.blue('Hello', 'World!', 'Foo', 'bar', 'biz', 'baz'));
 
// Nest styles
log(chalk.red('Hello', chalk.underline.bgBlue('world') + '!'));
 
// Nest styles of the same type even (color, underline, background)
log(chalk.green(
    'I am a green line ' +
    chalk.blue.underline.bold('with a blue substring') +
    ' that becomes green again!'
));

Output is shown below.

You can also easily define your own themes.

const chalk = require('chalk');
 
const error = chalk.keyword('red');
const warning = chalk.keyword('orange');
const success = chalk.keyword('green');
 
console.log(error('Error!'));
console.log(warning('Warning!'));
console.log(success('Done!'));

Chalk styles can be chained together, assuming the platform supports that particular style.

console.log(chalk.red.bold.underline('Hello', 'world'));

More detailed information can be found at the chalk nodejs npm.

The post Styling terminal string in NodeJS appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Many times we need to get a statistical distribution of values in a database table. Say you have a e-commerce shoe store having a product table with the following fields and values. As this is only an example I’ve limited the table to a few items; there will hundreds of rows in a real-life table.

;'product' table
;
+----+----------+-------+------+-------+
| id | type     | color | size | price |
+----+----------+-------+------+-------+
|  1 | sandals  | black |    7 | 45.00 |
|  2 | boots    | black |    8 | 36.00 |
|  3 | loafers  | brown |    8 | 45.00 |
|  4 | sandals  | brown |    7 | 45.00 |
|  5 | sneakers | blue  |    6 | 25.00 |
|  6 | sneakers | black |    6 | 25.00 |
|  7 | sandals  | blue  |    7 | 45.00 |
|  8 | loafers  | black |    7 | 35.00 |
|  9 | boots    | brown |    8 | 42.00 |
+----+----------+-------+------+-------+

What we want is to get a distribution which lists how many of each value we have got, like: sandals = 3, boots = 2, size 7 shoes = 4 etc.

The following query will accomplish that.

SELECT 'type', product.type, COUNT(*) AS total_rows 
    FROM product GROUP BY product.type 
UNION
SELECT 'color', product.color, COUNT(*) AS total_rows 
    FROM product GROUP BY product.color  
UNION
SELECT 'size', product.size, COUNT(*) AS total_rows 
    FROM product GROUP BY product.size  
UNION
SELECT 'price', product.price, COUNT(*) AS total_rows 
    FROM product GROUP BY product.price ;

This will return the following results.

+-------+----------+------------+
| type  | type     | total_rows |
+-------+----------+------------+
| type  | boots    |          2 |
| type  | loafers  |          2 |
| type  | sandals  |          3 |
| type  | sneakers |          2 |
| color | black    |          4 |
| color | blue     |          2 |
| color | brown    |          3 |
| size  | 6        |          2 |
| size  | 7        |          4 |
| size  | 8        |          3 |
| price | 25.00    |          2 |
| price | 35.00    |          1 |
| price | 36.00    |          1 |
| price | 42.00    |          1 |
| price | 45.00    |          4 |
+-------+----------+------------+

We now know how many of each type we have. This will be helpful to carryout any statistical analysis on the products table. We can create a new table (product_stats) with the above query by adding a ‘INSERT’ statement.

INSERT INTO product_stats (column_name, value, total_rows)
 
SELECT 'type', product.type, COUNT(*) AS total_rows 
    FROM product GROUP BY product.type 
UNION
SELECT 'color', product.color, COUNT(*) AS total_rows 
    FROM product GROUP BY product.color  
UNION
SELECT 'size', product.size, COUNT(*) AS total_rows 
    FROM product GROUP BY product.size  
UNION
SELECT 'price', product.price, COUNT(*) AS total_rows 
    FROM product GROUP BY product.price ;

The post Statistical distribution of column values in MySQL appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Recently I changed domain name of one of my other WordPress blogs. However after transferring I found that the GUId’s of the old posts were not updated. They were still referencing the old domain. This caused the feed of the new site to show incorrect domains. My initial thought was to to use a plugin, but it would have taken some time to search for the one I required. So the next best thing for me was to do a update on the post table directly from my hosting PHPMyAdmin interface.

A simple way to do it is to use the MySQL ‘REPLACE’ function to change th domain name in the GUID column in the ‘wp_posts’ table. The following code shows a way to accomplish the same. This will replace all GUID column values in the wp_posts table having ‘domain1.com’ as a url with ‘domain2.com’.

UPDATE wp_posts 
SET 
   guid = REPLACE(guid,
                  'domain1.com',
                  'domain2.com')
 
  where guid like '%domain1.com%'

The ‘REPLACE’ function takes three parameters :

REPLACE(original_string, string_to_replace, string_to_replace_with);

A example is given below.

SELECT REPLACE('https://www.domain1.com', 'domain1.com', 'domain2.com');
-> https://www.domain2.com

So in our above example we are replacing the ‘domain1.com’ string with the ‘domain2.com’ string. Do not confuse REPLACE function with the REPLACE statement, both are different.

Note that when searching for text to replace, MySQL uses case-sensitive match to perform a search for a string to be replaced.

Also it very important that the first parameter to REPLACE here is the column name
and not a pure string, so DO NOT put quotes. If you put quotes to the field name like “guid”, the query will replace the content of that column with the string “guid”.

Also as usual whenever making any changes to the database, always make a backup.

So in conclusion, as I was already at the installation stage and had my hosting cPanel open, this solution was the quickest for me.

The post Updating WordPress GUIDS appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Code-diesel Blog by Sameer - 7M ago

Tuples are one of the core object types in Python. Although tuples may seem like lists, they are often used in different situations and for different purposes and unlike lists they are immutable.

Syntactically they are enclosed in parenthesis (or without them) instead of square brackets like lists, but they support arbitrary types and the usual operations available with lists.

>>> tuple = (123, 'Hello World!', 21.03, 'AAA')
>>> tuple
(123, 'Hello World!', 21.03, 'AAA')
>>> len(tuple)
4
>>> tuple + ('BBB', 'CCC')
(123, 'Hello World!', 21.03, 'AAA', 'BBB', 'CCC')
>>> tuple
(123, 'Hello World!', 21.03, 'AAA')
>>> tuple[0]
123
>>> tuple[1]
'Hello World!'
>>> tuple[0] = 10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

If you look at the last line above you can see that changing of a tuple value is not permitted as tuples are immutable. That is the main difference between tuples and lists in Python.

The reason to use tuples is to provide a integrity in large programs where you want to make sure that some data structures are not changed accidentally in the program. If you want to design a data structure and do not want to accidentally change the original data, tuples are what you should use.

The post Tuples in Python appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Lists are a versatile Python data structure and allows various sequence operations. Besides the usual, Lists also include a powerful operation called as ‘list comprehension’ which allows us to process various list structures, like a matrix or nested arrays or lists. It allows you to concisely form a new list by filtering the elements of a list.

List comprehension takes the following form.

[expr for val in collection if condition]

Which is equivalent to the following for loop.

result = []
for val in collection:
   if condition:
      result.append(expr)

Take the example matrix given below.

>>> matrix = [[1,2,3],
              [4,5,6],
              [7,8,9]]
>>> matrix
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

It is easy to get rows from the matrix as they stored in a row format, like given below.

>>> matrix[0]
[1, 2, 3]
>>> matrix[1]
[4, 5, 6]
>>> matrix[2]
[7, 8, 9]

But what if you wanted to grab the columns from the matrix. Here we can use list comprehensions. List comprehensions are a way to build new lists by running a expression on each element in the sequence.

>>> col1 = [row[0] for row in matrix]
>>> col1
[1, 4, 7]
>>> col2 = [row[1] for row in matrix]
>>> col2
[2, 5, 8]
>>> col3 = [row[2] for row in matrix]
>>> col3
[3, 6, 9]

The first line above basically asks to give us the item at location row[0] (first item in the row) for each row in the matrix which is than stored in the new ‘col1’ list. So for example on the first iteration of the matrix row[0] will return ‘1’, on the second iteration it will return ‘4’ and lastly ‘7’.

We can also make list comprehensions more complex.

# Here we have added 1 to each column element before storing in the new list
 
>>> col1_new = [row[0] + 1 for row in matrix]  
>>> col1_new
[2, 5, 8]
# Double each character found in the string and create a new list.
 
>>> [c * 2 for c in 'Hello']
['HH', 'ee', 'll', 'll', 'oo']

Note that list comprehensions has to be in square brackets which basically hints that a list is returned. The following for example will not work.

>>> col1 = row[0] for row in matrix
  File "<stdin>", line 1
    col1 = row[0] for row in matrix
                    ^
SyntaxError: invalid syntax

You can also add a filter condition to the expression.

# Get only those elements which are greater than 1
 
>>> [row[0] for row in matrix if row[0] > 1]
[4, 7]
 
 
# Get only numbers which are even
 
>>> [row[1] for row in matrix if row[1] % 2 == 0]
[2, 8]
>>> [row[0] for row in matrix if row[0] % 2 == 0]
[4]

This is just a introductory article around list comprehensions, as they are more powerful in what they can accomplish. We will cover them more in future articles.

The post List Comprehensions in Python appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Since I started working with computers, my favorite file format was ‘zip’, popularized by the Windows PKZIP program. The idea of compressing data into shorter versions of themselves was a fascinating idea.

Then, I was ignorant of the mathematical foundations of information theory, so it seemed like magic. I spent umpteen hours playing with file compression – exploring the various options available and seeing how much further I could compress a file. Now compression is a normal part of any computer language library. So when learning Python I first started exploring the compression library.

The Python ‘zipfile’ module can be used to read and write zip files. A simple exampe to view the contents of a zip file is given below.

from __future__ import print_function
import zipfile
 
with zipfile.ZipFile('python.zip','r') as zf:
        print(zf.namelist())
 
# Returns a list of file names in the zip file
# ['test.py', 'test2.py', 'test3.py']

We can test if file is a zip file regardless the extension.

>>zipfile.is_zipfile('python.zip')
>>True

Also we can test for multiple files.

from __future__ import print_function
import zipfile
 
for filename in ['python.zip', 'test.py', 'test2.py']:
        print('{:>20}  {}'.format(filename, zipfile.is_zipfile(filename)))

Which will return.

python.zip  True
   test.py  False
  test2.py  False

To extract all the files from the zip archive do the following. This will extract all the file to the current directory.

>>import zipfile
>>zp = zipfile.ZipFile('python.zip')
>>zp.extractall()

You can also specify the path.

>>zp.extractall('./test)

We can also extract a single file from the zip archive by specifying the file name.

>>zp.extract('test.py')

This is just a short overview of the zipfile module. In future posts we will go into other details.

The post Working with zip files in Python appeared first on codediesel.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Connecting to MySQL from Python offers many solutions, but I like the official MySQL connector. You can install the module using pip.

sudo pip install mysql-connector-python

Once the module is installed you can connect to MySQL using the following code.

import mysql.connector
 
cnx = mysql.connector.connect(user='mysql_username',
                              password='mysql_password',
                              host='localhost',
                              database='database_name')
cursor = cnx.cursor()
 
# Example query to connect to the MySQL database and user table and user column.
query = "SELECT user FROM user"
cursor.execute(query)
 
# Iterate over the resulting cursor and print
for user in cursor:
    print(user)
 
cursor.close()
cnx.close()

More detailed information can be found here.

The post How to connect to MySQL in Python appeared first on codediesel.

Read Full Article

Read for later

Articles marked as Favorite are saved for later viewing.
close
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Separate tags by commas
To access this feature, please upgrade your account.
Start your free month
Free Preview