Loading...

Follow SDET.US | Golang on Feedspot

Continue with Google
Continue with Facebook
or

Valid
Concurrency

The below video was shared on Twitter.

It’s a really great view into concurrency and how Go’s system differs from “Threads” in other languages:

GoWayFest 2017 - Eyal Post - Why Go Scales? - YouTube

GoRoutines vs. Threads

At about 12:44 into the video Eyal gets into the methodology of Go concurrency (Go Routines) and how they differ from threads.

The memory stack of a Goroutine is very small. It starts at 2kb.

The go stack grows as needed and it also shrinks when no longer needed.

If 10,000 goroutines were created, they would only consume 20MB of memory.

The above values are much smaller than the stack size of threads.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Back when I was taking the Offensive Security course… we made a lot of use of Kali Linux’s “dirb” utility… it was a brute force utility of a directory structure on a domain.  It would take a dictionary file, and run each word in the dictionary, appended to the domain.  For example:

If the domain was mydomain.xxx it would try:
mydomain.xxx/a

mydomain.xxx/admin

and so on. Each time dirb got a 200 OK, it listed it as a valid endpoint.

Dirgo

Dirgo was a fun project in which I tried to emulate some of this behavior and add a small twist to dirb.  Where dirb looks for a valid response on a URL, I built Dirgo to let the user specify the response they are looking for.

For example, maybe you don’t care to find what reports a 200, perhaps you prefer to find what responds with a 500 server error… This is a great little utility for QA’ing a web app.  Sometimes a view / endpoint is created but it doesn’t respond well when called on its own.  Or you may have endpoints that just are buggy and no one reported they were going live.  You could add endpoints to the dictionary, or try and find them on your own.

In my own testing with  Dirgo I found several endpoints on servers at work where we have 500’s reporting, that should render fine.  All found using the common.txt dictionary.

Code

Below is the code… it’s super simple. Nothing complex here.

package main
import(
	"bufio"
	"fmt"
	"os"
	"net/http"
	"strings"
)

var host string
var statusCode string

func main(){
	userInput()
	words(host)
}

func userInput(){
	fmt.Println("Host (include http(s)://> ")
	fmt.Scan(&host)
	fmt.Println("Status code (i.e. 200)> ")
	fmt.Scan(&statusCode)
}

func words(url string){
	file, ferr := os.Open("dictionaries/common.txt")
	if ferr != nil{
		fmt.Println(ferr)
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)

	for scanner.Scan() {
		enumerate(url, scanner.Text())
	}
	fmt.Println("Done...")
}

func enumerate(url, word string){
	conn, err := http.Get(url + "/" + word)
	if err != nil {
		fmt.Println(err)
	}
	if strings.Contains(conn.Status, statusCode){
		fmt.Printf("%s%s%s%s%s%s",url, "/", word, " : ", statusCode,"\n")
	}
}
Future Enhancements
  • I would like to make the dictionary something passed on the command line… rather than building it each time with a different dictionary.
  • I also would like to figure out some gentle concurrency for this project (running sections of the dictionary file concurrently.)
  • Add more of the feature set of dirb: https://tools.kali.org/web-applications/dirb
Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

In my work, I write code on a Mac and deploy to work environments like Windows Server or Centos Linux.

So I needed to know how to build cross platform.  Running “go build” just built for my local machine.

I came across a video dealing with the subject:

Cross Compiling Go Lang for multiple platforms - YouTube

In my case, I didn’t follow the entirety of his advice.  I had already installed XCode in the past, so I didn’t need to jump through those hoops.

All I needed was to run these params on the build command line, which set the OS and Architecture for the target device:

GOOS=linux GOARCH=amd64 go build portScanner.go

The above example sets the build process to make a binary for the Linux OS using the amd64 bit chipset.

I was able to do the same for Windows.  I’m not sure of all the values available.  I noticed I couldn’t specify intel64 or intel32 or x86 for Windows.  In my app, I was able to specify amd64 for Windows (even though it was an intel chipset) and the exe binary worked fine.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Each thing I learn in Go, I find myself trying to refer to it as I would in another language (thinking of structs as Classes, for example.)

There are however some differences with the Go implementation of these concepts.  In this post I wanted to take some material I’ve been reading about and learning about in various courses and books.  Specifically, I’d like to refer to the concept of concurrency.

GoRoutines

A goroutine is a way of running functions concurrently.  In Grails there is a concept of using “promises” and tasks… and this is kinda similar.  But keep in mind that concurrent doesn’t necessarily mean parallel.

In my Port Scanner, I was originally running each check on a port in a range sequentially.  It would take several min to iterate over a list of 8000 ports.  However, when I applied the concept of GoRoutines, I got this down to 20 seconds to run over 8,000 localhost ports.

Setting Up GoRoutines

To make use of GoRoutines we need to define a wait group.  The concept of a WaitGroup works like this:

  1. we define a wait group
  2. we define how many groups (concurrent functions) we’ll be doing
  3. we call the function using the “go” keyword (defining it as a Go Routine)
  4. Once the function process is done, we mark the group as done.

In practicality it would look like this:

var wg sync.WaitGroup
wg.Add(4)		// 3s to run 1000 ports on 4 concurrent groups
	go check_port(host, start_port, end_port_set1)
	go check_port(host, (end_port_set1 + 1), end_port_set2)
	go check_port(host, (end_port_set2 + 1), end_port_set3)
	go check_port(host, (end_port_set3 + 1), end_port)
	wg.Wait()

func check_port(host string, start_port, end_port int) {

	for i := start_port; i <= end_port; i++ {
		//fmt.Println('\n')
		qualified_host := fmt.Sprintf("%s%s%d", host, ":", i)
		conn, err := net.DialTimeout("tcp", qualified_host, 10*time.Millisecond)  // Got the timeout code from: https://stackoverflow.com/questions/37294052/golang-why-net-dialtimeout-get-timeout-half-of-the-time
		if err != nil {
			continue
		}
		fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n1\n22\n\n\n\n")
		conn.SetReadDeadline(time.Now().Add(10*time.Millisecond))

		// swapping bufio reads to reading buffers as bytes gets huge performance increase: 8000 ports in 20s (vs 1min 10s using bufio reads)
		buff := make([]byte, 1024)
		n, _ := conn.Read(buff)
		fmt.Printf("Port: %d%s\n",i, buff[:n])
	}
	wg.Done()
}

The first line defines the variable wg to hold the value for the WaitGroup method of the sync package (which will need to be imported.)

wg then makes use of the built in method Add() and in this case I put in the value of 4 wg.Add(4) which will run 4 concurrent runs.

Each concurrent call is defined as:

go [function name](params)

In this case:

go check_port(host, start_port, end_port_set1)…

At the bottom of this list of calls there’s a wait invoked: wg.Wait()

Lastly, the function itself has a wg.Done() at the end of the logic.

Channels in Go

Channels are a complex subject for me.  At first, I thought it was the magic that made concurrency happen in Go. Instead, I discovered that GoRoutines are how concurrency is applied in a go app.  Channels offer the ability for GoRoutines to share data back and forth.

I initially thought of channels as a “Thread,” as in other languages.

However, in reading this programmer’s blog post, I discovered some differences to threads: http://tleyden.github.io/blog/2014/10/30/goroutines-vs-threads/

You can see their full bulleted list of differences there.  I’ll just mention a couple highlights:

  1. GoRoutines can be run at a factor more than Threads on a system.
  2. GoRoutines have a mechanism to share and talk between GoRoutines (channels.)

Channels are mechanisms to share data between GoRoutines.

More info on channels:

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
SDET.US | Golang by Bwarner - 7M ago

Notes from the course Learn How to Code: Google’s Go Programming Language

Go and Types

Like all languages values are of certain types.  Types can be int, string, boolean, etc.  Within Go, you can also define your own types.

type hacker bool

func main(){
	var eliot hacker = true
	fmt.Println(eliot)
	fmt.Printf("%T ", eliot)
}

In the above example, eliot is set to a new type called “hacker.”  As we know in Mr. Robot, Eliot is a hacker, so this type has the underlying type of boolean.

In the main function, eliot is set to true.  Verifying this and its type is done with the Print statements, which output:

true
main.hacker 
OOP and Go

In a post from 2013, Bill Kennedy writes:

I consider Go to be a light object oriented programming language. Yes it does have encapsulation and type member functions but it lacks inheritance and therefore traditional polymorphism. For me, inheritance is useless unless you want to implement polymorphism. With the way interfaces are implemented in Go, there is no need for inheritance. Go took the best parts of OOP, left out the rest and gave us a better way to write polymorphic code.  –From Goinggo.net

If you search long enough, you’ll soon discover that this topic is covered (and debated) quite a bit.  Yes, Go is Object Oriented, but it lacks the traditional elements we associate with Object Oriented Languages (like classes, inheritance, etc.)

In Todd McLeod’s course on Go [link] he states that Go is OOP with these elements you would find in other OOP languages:

  • Encapsulation
  • Reusability through inheritance or embedded types
  • Polymorphism through interfaces
  • Overriding through promotion

The above seems a tad contrary to some of the other links and discussion that I’ve come across.  However, Todd makes a good argument for these points.

Regarding Encapsulation, Todd states that by creating our own types and that type has fields that hold state.  Those types can be associated to behaviors (like methods.)

For Inheritance, Todd adequately shows in the course that a type can be injected anonymously into code.  When done, all the fields that were part of the struct are carried (promoted) with the injection.

Example:

type Car struct {
	make string
	model string
	year int
}

type Demolition struct {
	Car
}

As for Polymorphism, Todd McLeod explains that interfaces fit the bill.

Finally, in regards to Overriding methods/functions, Todd McLeod shows that through “promotion” a function can be overridden.  This is a complex subject for me.  However, in watching and rewatching the lecture several times I have an idea of what he’s getting at. Below is a code example I wrote, that follows his own:

type Car struct {
	Make string
	Model string
	Year int
}

type Demolition struct {
	Car
	Make string
	Convertable bool
}

func main(){
	mustang := Demolition{
		Car: Car{
			Make: "Ford",
			Model: "Mustang",
			Year: 1973,
		},
		Make: "FORD MOTORS",
		Convertable: true,
	}

	fmt.Println(mustang.Make, mustang.Model,  mustang.Year, mustang.Car.Make)
}

In the above code the output will be:

FORD MOTORS Mustang 1973 Ford

mustang.Make prints out “FORD MOTORS” and mustang.Car.Make will return the value “Ford.”  What occurred here is that the Car struct was promoted into the Demolition struct and and when we set a value to Demotion, we created a set of values for the mustang car.  However, another definition of “Make:” was created which shows overriding the initial value.

Structs in Go

The reason why I started with the concept of Types is that the instructor (Todd McLeod) brought up the idea first before going into Structs.  Here’s why – When making Structs in Go, you define a type of struct.

A struct is a data set that resembles (at a glance) something akin to a class in other languages.  Example:

type car struct {
	make string
	model string
	year int
}

func main(){
	mustang := car{"Ford", "Mustang", 1973}
	fmt.Println(mustang.make, mustang.model,  mustang.year)
}

The value mustang is of type car.  It is reminiscent of a class being instantiated as objects.  I’m not sure if that’s what’s actually happening… and my understanding of the deep waters of OOP is quite limited.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Note: Only utilize a port scanner on sites you have permission to test. Also, note the legalities of port scanners in your territories.

Video Demo

Learning Golang - Making a concurrent Port Scanner in Google's Go Language - YouTube

Go Project

As I’m learning the Go language, I thought I’d work on a small project of making a utility I wrote in Python, in the Go language.  In this case, I chose a port scanner I wrote in Python.  The scanner tries to interrogate ports and return any banner that would indicate what’s installed there.

The original code I wrote in Python utilized the socket and sys libraries.  I wrote it while taking an InfoSec course over at Offensive Security.  Anyway… years later, I’m thinking of a project to try and do in Go (just go kick the tires and get used to the code) and I figure I might as well port that project into Go.

Go Port Scanner

The code is simple it basically kicks off a shell prompt asking for a host, then a starting port and an ending port.  It then iterates over the port range, trying each port in turn.

Here’s the code (link):

// +build linux, 386, darwin

package main

import (
	"fmt"
	"net"
	"time"
	"sync"
)

var host string
var start_port int
var end_port int
var wg sync.WaitGroup

func main() {
	user_input()
}

func check_port(host string, start_port, end_port int) {

	for i := start_port; i <= end_port; i++ {
		//fmt.Println('\n')
		qualified_host := fmt.Sprintf("%s%s%d", host, ":", i)
		conn, err := net.DialTimeout("tcp", qualified_host, 10*time.Millisecond)  // Got the timeout code from: https://stackoverflow.com/questions/37294052/golang-why-net-dialtimeout-get-timeout-half-of-the-time
		if err != nil {
			continue
		}
		fmt.Fprintf(conn, "GET / HTTP/1.0\r\n\r\n1\n22\n\n\n\n")
		conn.SetReadDeadline(time.Now().Add(10*time.Millisecond))

		buff := make([]byte, 1024)
		n, _ := conn.Read(buff)
		fmt.Printf("Port: %d%s\n",i, buff[:n])
	}
	wg.Done()
}

func user_input() {
	fmt.Println("Host> ")
	fmt.Scan(&host)
	fmt.Println("Starting Port (i.e. 80)> ")
	fmt.Scan(&start_port)
	fmt.Println("End Port (i.e. 8080)> ")
	fmt.Scan(&end_port)
	fmt.Println("Running scan... ")

	port_range := end_port - start_port
	end_port_set1 := (port_range / 10) + start_port
	end_port_set2 := (port_range / 10) + end_port_set1
	end_port_set3 := (port_range / 10) + end_port_set2
	end_port_set4 := (port_range / 10) + end_port_set3
	end_port_set5 := (port_range / 10) + end_port_set4
	end_port_set6 := (port_range / 10) + end_port_set5
	end_port_set7 := (port_range / 10) + end_port_set6
	end_port_set8 := (port_range / 10) + end_port_set7
	end_port_set9 := (port_range / 10) + end_port_set8


	wg.Add(10)		// 1min to run 65,000 ports on 10 concurrent groups
	go check_port(host, start_port, end_port_set1)
	go check_port(host, (end_port_set1 + 1), end_port_set2)
	go check_port(host, (end_port_set2 + 1), end_port_set3)
	go check_port(host, (end_port_set3 + 1), end_port_set4)
	go check_port(host, (end_port_set4 + 1), end_port_set5)
	go check_port(host, (end_port_set5 + 1), end_port_set6)
	go check_port(host, (end_port_set6 + 1), end_port_set7)
	go check_port(host, (end_port_set7 + 1), end_port_set8)
	go check_port(host, (end_port_set8 + 1), end_port_set9)
	go check_port(host, (end_port_set9 + 1), end_port)
	wg.Wait()

}
Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

In the Go language there is a concept of deferring an action till later.

As far as I can tell, the usefulness of this is to allow for the code to be more readable.   In Todd McLeod’s course on Go, he offers an example of this used in an application he wrote. In the app a file is opened, some actions are done to the opened file (such as writing content to it) and then the file is closed.  Instead of having the close at the bottom of the function, he has the close call under the open call.

Example:

ipfile, err := os.Open(“ipaddresses.txt”)

defer ipfile.Close()

In the above example, the close is near the call to open the file.  The work done on the file is below the defer action.  Before the function closes, it runs the defer actions

As far as I know… this helps in readability (i.e. we put the open and close actions together), but has no real functionality impact as this could easily have been done using the close call at the end of the function.

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

I hadn’t come across this concept in previous languages I worked in.  In Go, however, this concept came up in an online course I was taking on Udemy.com (from Todd McLeod.)  He raised the concept of a Variadic Function.

Variadic Functions

These are functions which take an unspecified amount of parameters.  In a regular function you would lock down what parameters are allowed (how many and of what types.)  In a Variadic Function, you use “…” notation to allow for an undefined amount values to get passed.

Languages Using Variadic Functions

Go is not the only language to use Variadic Functions.  Other languages that make use of this (even using the same triple dot notation) are:

  • C++
  • Java
  • JavaScript

Python and Ruby also make use of this concept, although with a different notation (the asterisk.)

Examples of Variadic Functions in Go

Here’s some code showing a variadic function in Golang:

package main
import "fmt"

func main(){
	variadicsf("Brian", "Austin", "Lisa", "Travis", "Dan")
}

func variadicsf(names ...string){
	fmt.Println(names)

	for _, name := range names {
		fmt.Println(name)
	}
}

// The Variadic function allows for multiple params to be passed in.
// Notation of this is in the form of ...
// These params are put into an array.
// Also note the for loop making use of a blank identifier (the underscore)
// The blank identifier allows me to ignore the key of the array and only concern myself with the value.

I the above example, the code will output the collection/array (fmt.Println(names)) as well loop through the array and output each name passed in.  Here’s the outuput:

[Brian Austin Lisa Travis Dan]
Brian
Austin
Lisa
Travis
Dan

Other Variadic Forms

It is possible to have a variable function and you want to pass into it data from a slice.  The data could be like this (taken from Todd McLeod’s course): data := []float64{43, 24,10,24,33,54}

That data can not simply be passed into a variadic function.  Instead, the call to the function needs to be modified like so:

multip(data…)

func multip(num …float64){ }

What is occurring in this situation is that the call to the method is using a variadic parameter (triple dots appended to the param.)  This sends each element to the function, one at a time.  So first 43 is sent into the multip function, then 24, then 10 and so on.

For more info on these topics, check out Todd McLeod’s course: Learn to Code using Go

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

As a little example of basic Golang, I wrote a little app that takes user input and reverses the word as output:

package main

import "fmt"

var userInput string

func main() {
   reverseWord()
}

func reverseWord() {

   fmt.Println("Input a word> ")
   fmt.Scan(&userInput)
   var word_length int = len(userInput)
   for i := word_length; i > 0; i-- {
      fmt.Print(userInput[(i - 1):i])
   }
}

Read Full Article
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 
Unused Code in Go (Golang)

In the Go language (Golang) there is a concept that threw me for at first.  In other languages, you can have unused code sitting around.  By unused I don’t mean comments, instead I mean code that isn’t actively being consumed by the application.  For example, if I declared two variables (a and b) but only initialized variable a, even though the variable assignment for b is valid – it will not compile, as variable b is not getting consumed.

This language requirement keeps the code useful.

Blank Identifiers in Go (Golang)

The underscore is the blank identifier.  It is used to stand in place of a variable assignment. For example, if you had a call to http.Get you’ll get back the response or an error.  Instead of writing the catch code to handle an error, you could call:

resp, _ : = http.Get(….)

Read Full Article

Read for later

Articles marked as Favorite are saved for later viewing.
close
  • Show original
  • .
  • Share
  • .
  • Favorite
  • .
  • Email
  • .
  • Add Tags 

Separate tags by commas
To access this feature, please upgrade your account.
Start your free month
Free Preview