1 2 3 Previous Next

manning_pubs

89 posts
By Christopher W. H. Davis, Continuous Improvement 

Save 40% on Continuous Improvement with discount code jn15ci at manning.com.

The book Continuous Improvement walks the reader through the collection and analysis of data to provide metrics that guide the continuous improvement of agile teams. Each chapter ends in a case study outlining the application of this technique with specific types of data. This is an excerpt of a real world scenario from the end of Chapter 4 that shows the reader how to apply source control data toward continuous improvement. We

By Brian Ketelsen, Erik St. Martin, and William Kennedy

Save 40% on Go in Action with discount code jn15goia at manning.com.

The purpose of the work package is to show how you can use an unbuffered channel to create a pool of goroutines to perform and control the amount of work that gets done concurrently. This is a better approach that using a buffered channel of some arbitrary static size that acts as a queue of work and throwing a bunch of goroutines at it.

Unbuffered channels provide a guarantee that data has been exchanged between two goroutines. The approach this package takes by using an unbuffered channel allows the user to know when the pool is performing the work and pushes back when it can't accept any more work because it is busy. No work is ever lost or stuck in queue that has no guarantee it will ever be worked on.

Let's take a look at the

work.go
code file from the work package:

 

Listing 1

chapter7/patterns/work/work.go
01 // Example provided with help from Jason Waldrip.
02 // Package work manages a pool of goroutines to perform work.
03 package work
04
05 import "sync"
06
07 // Worker must be implemented by types that want to use
08 // the work pool.
09 type Worker interface {
10  Task()
11 }
12
13 // Pool provides a pool of goroutines that can execute any Worker
14 // tasks that are submitted.
15 type Pool struct {
16  work chan Worker
17  wg sync.WaitGroup
18 }
19
20 // New creates a new work pool.
21 func New(maxGoroutines int) *Pool {
22  p := Pool{
23      tasks: make(chan Worker),
24  }
25
26  p.wg.Add(maxGoroutines)
27  for i := 0; i < maxGoroutines; i++ {
28      go func() {
29          for w := range p.work {
30              w.Task()
31          }
32          p.wg.Done()
33      }()
34  }
35
36  return &p
37 }
38
39 // Run submits work to the pool.
40 func (p *Pool) Run(w Worker) {
41  p.work <- w
42 }
43
44 // Shutdown waits for all the goroutines to shutdown.
45 func (p *Pool) Shutdown() {
46  close(p.tasks)
47  p.wg.Wait()
48 }

The work package in listing 2 starts off with the declaration of an interface named

Worker
and a struct named 
Pool
:

 

Listing 2

07 // Worker must be implemented by types that want to use
08 // the work pool.
09 type Worker interface {
10  Task()
11 }
12
13 // Pool provides a pool of goroutines that can execute any Worker
14 // tasks that are submitted.
15 type Pool struct {
16  work chan Worker
17  wg sync.WaitGroup
18 }
The 
Worker
interface in listing 1 on line 10 declares a single method called 
Task
. Then on line 15, a struct named 
Pool
is declared which is the type that implements the pool of goroutines and will have methods that process the work. The type declares two fields, one named 
work
which is a channel of the 
Worker
interface type and a 
sync.WaitGroup
named 
wg

Next, let's look at the factory function for the work package:

Listing 3

20 // New creates a new work pool.
21 func New(maxGoroutines int) *Pool {
22  p := Pool{
23      work: make(chan Worker),
24  }
25
26  p.wg.Add(maxGoroutines)
27  for i := 0; i < maxGoroutines; i++ {
28      go func() {
29          for w := range p.work {
30              w.Task()
31          }
32          p.wg.Done()
33      }()
34  }
35
36  return &p
37 }

Listing 3 shows the

New
function that is used to create work pool that is configured with a fixed set number of goroutines. The number of goroutines is passed in as a parameter to the 
New
function. On line 22, a value of type 
Pool
is created and the 
work
field is initialized with an unbuffered channel.

 

Then on line 26, the

WaitGroup
is initialized and on lines 27 through 34 the specified number of goroutines are created. The goroutine just receives interface values of type 
Worker
and calls the Task method on those values:

 

Listing 4

28         go func() {
29          for w := range w.work {
30              w.Task()
31          }
32          p.wg.Done()
33      }()

The

for range
loop blocks until there is a 
Worker
interface value to receive on the work channel. When a value is received, the 
Task
method is called. Once the 
work
channel is closed, the 
for range
loop ends and the call to 
Done
on the 
WaitGroup
is called. Then the goroutine terminates.

 

Now that we can create a pool of goroutines that can wait for and execute work, let's look at how work is submitted into the pool:

Listing 5

39 // Run submits work to the pool.
40 func (p *Pool) Run(w Worker) {
41  w.work <- w
42 }

Listing 5 shows the

Run
method. This method is used to submit work into the pool. It accepts an interface value of type 
Worker
and sends that value through the work channel. Since the 
work
channel is an unbuffered channel, the caller must wait for a goroutine from the pool to receive it. This is what we want because the caller needs the guarantee that the work being submitted is being worked on once the call to 
Run
returns.

 

At some point the work pool need to be shutdown. This is where the

Shutdown
method comes in:

 

Listing 6

44 // Shutdown waits for all the goroutines to shutdown.
45 func (p *Pool) Shutdown() {
46  close(p.work)
47  p.wg.Wait()
48 }

The

Shutdown
method in listing 6 does two things. First, it closes the 
work
channel which causes all of the goroutines in the pool to shut down and call the 
Done
method off the 
WaitGroup
. Then the 
Shutdown
method calls the 
Wait
method on the 
WaitGroup
which causes the 
Shutdown
method to wait for all the goroutines to report they have terminated.

 

Now that we have seen the code for the

work
package and learned how it works, let's review the test program in the 
main.go
source code file:

 

Listing 7

chapter7/patterns/work/main/main.go

01 // This sample program demonstrates how to use the work package
02 // to use a pool of goroutines to get work done.
03 package main
04
05 import (
06  "log"
07  "sync"
08  "time"
09
10  "github.com/goinaction/code/chapter7/patterns/work"
11 )
12
13 // names provides a set of names to display.
14 var names = []string{
15  "steve",
16  "bob",
17  "mary",
18  "therese",
19  "jason",
20 }
21
22 // namePrinter provides special support for printing names.
23 type namePrinter struct {
24  name string
25 }
26
27 // Task implements the Worker interface.
28 func (m *namePrinter) Task() {
29  log.Println(m.name)
30  time.Sleep(time.Second)
31 }
32
33 // main is the entry point for all Go programs.
34 func main() {
35  // Create a work pool with 2 goroutines.
36  p := work.New(2)
37
38  var wg sync.WaitGroup
39  wg.Add(100 * len(names))
40
41  for i := 0; i < 100; i++ {
42      // Iterate over the slice of names.
43      for _, name := range names {
44          // Create a namePrinter and provide the
45          // specific name.
46          np := namePrinter{
47              name: name,
48          }
49
50          go func() {
51              // Submit the task to be worked on. When RunTask
52              // returns we know it is being handled.
53              p.Run(&np)
54              wg.Done()
55          }()
56      }
57  }
58
59  wg.Wait()
60
61  // Shutdown the work pool and wait for all existing work
62  // to be completed.
63  p.Shutdown()
64 }

Listing 7 shows the test program that uses the package work to perform the displaying of names. The code starts out on line 14 with the declaration of a package level variable named

names
which is declared as a slice of strings. The slice is also initialized with five names. Then a type named 
namePrinter
is declared:

 
22 // namePrinter provides special support for printing names.
23 type namePrinter struct {
24  name string
25 }
26
27 // Task implements the Worker interface.
28 func (m *namePrinter) Task() {
29  log.Println(m.name)
30  time.Sleep(time.Second)
31 }

On line 23 in listing 7, the type

namePrinter
is declared and the implementation of the 
Worker
interface follows. The purpose of the work is to display names to the screen. The type contains a single field named 
name
which will contain the name to display. The implementation of the 
Worker
interface uses the 
log.Println
function to display the name and then waits a second before returning. The second wait is just to slow the test program down so we can see the concurrency is action.

 

With the implementation of the

Worker
interface, we can look at the code inside the main function:

 
33 // main is the entry point for all Go programs.
34 func main() {
35  // Create a work pool with 2 goroutines.
36  p := work.New(2)
37
38  var wg sync.WaitGroup
39  wg.Add(100 * len(names))
40
41  for i := 0; i < 100; i++ {
42      // Iterate over the slice of names.
43      for _, name := range names {
44          // Create a namePrinter and provide the
45          // specific name.
46          np := namePrinter{
47              name: name,
48          }
49
50          go func() {
51              // Submit the task to be worked on. When RunTask
52              // returns we know it is being handled.
53              p.Run(&np)
54              wg.Done()
55          }()
56      }
57  }
58
59  wg.Wait()
60
61  // Shutdown the work pool and wait for all existing work
62  // to be completed.
63  p.Shutdown()
64 }

On line 36 in listing 7, the New function from the

work
package is called to create the 
work
pool. The number 2 is passed into the call indicating the pool should only contain two goroutines. Then on lines 38 and 39 a 
WaitGroup
is declared and initialized to each goroutine that is going to be created. In this case, a goroutine for each name in the names slice is going to be created 100 times. This is to create a lot of goroutines competing to submit work to the pool.

 

Then on line 41 and 43, inner and outer

for
loops are declared to create all the goroutines. Inside each iteration of the inner loop, a value of type 
namePrinter
is created and provided with a name to print. Then on line 50, an anonymous function is declared and created as a goroutine. The goroutine calls the 
Run
method against the work pool to submit the 
namePrinter
value to the pool. Once a goroutine from the work pool receives the value, the call to 
Run
returns. This is turn causes the goroutine to decrement the 
WaitGroup
count and terminate.

 

Once all the goroutines are created, the

main
function calls 
Wait
on the 
WaitGroup
. The function will wait until all the goroutines that were created submit their work. Once 
Wait
returns, the 
work
pool is shut down by calling the 
Shutdown
method. This method will not return until all the work is complete. In our case, there would only be two outstanding pieces of work by this time.  
By Konstantinos Kapelonis, Java Testing with Spock

Save 40% on Java Testing with Spockwith discount code jn15spock at manning.com.

In the Java world, there has been so far only one solution for unit tests. The venerable JUnit framework is the obvious choice and has become almost synonymous with unit testing. JUnit has the largest mind share among developers who are entrenched in their traditions and don't want to look any further.

Even TestNG which has several improvements and is also fully compatible with JUnit has failed to gain significant traction.

But fear not! A new testing solution is now available in the form of Spock. Spock is a testing framework written in Groovy but able to test both Java and Groovy code. It is fully compatible with JUnit (it actually builds on top of the JUnit runner) and provides a cohesive testing package that also includes mocking/stubbing capabilities

It is hard to compare JUnit and Spock in a single article, because both tools have a different philosophy when it comes to testing. JUnit is a Spartan library that provides the absolutely necessary thing you need to test and leaves additional functionality (such as mocking and stubbing) to external libraries.

Spock has a holistic approach, providing you a superset of the capabilities of JUnit, while at the same time reusing its mature integration with tools and developments environments. Spock can do everything that JUnit does and more, keeping backwards compatibility as far as test runners are concerned.

What follows is a brief tour of some Spock highlights.

Writing concise code with Groovy syntax

Spock is written in Groovy which is less verbose than Java. This means that Spock tests are more concise than the respective JUnit tests. Of course this advantage is not specific to Spock itself. Any other Groovy testing framework would probably share this trait. But at the moment only Spock exists in the Groovy world.

Here is the advantage in a visual way, shown in figure 1.

 

Figure 1 Less code is easier to read, easier to debug, and easier to maintain in the long run.

Mocking and Stubbing with no external library

JUnit does not support Mocking and Stubbing on its own. There are several Java framework that fill this position. This is the main reason that I got interested in Spock in the first place is the fact that it comes full batteries included as mocking and stubbing are supported out of the box.

 

Figure 2 Spock is a superset of Junit

I

By Stephan Hochhaus and Manuel Schoebel, excerpted from Meteor in Action 

Save 40% on Meteor in Action with discount code jn15met at manning.com.

Oftentimes usernames and passwords are not the only option you want to give your users to log into an application. Being able to use an existing account to log into a site lowers the barrier of signing up by not having to type in a single bit of information. Additionally it simplifies using an application by not having to remember additional usernames or passwords.

 

Meteor ships with multiple authentication providers that allow users to use a social network instead of a local username. These networks include:

  • Facebook
  • Github
  • Google
  • Meetup
  • Meteor Developer Account
  • Twitter
  • Weibo

All of the above are based on oAuth, a complex way to pass authentication data from one site to another. There are also many community packages available that enable other authentication providers such as LinkedIn or Dropbox. As the fundamentals of working with Auth providers are the same for each provider, we will not need to discuss each.

An introduction to oAuth

The Open Authentication

By Debasish Ghosh, Functional and Reactive Domain Modeling 

Save 40% on Functional and Reactive Domain Modeling with discount code jn15frdmat manning.com.

One of the most important aspects of domain model API design is the specification of the API itself. The specification should be concise, clear and easily understandable by the client. While using object oriented techniques we use interfaces as the contract for API design. Interface based development is one of the primary techniques that

By Mala Gupta, OCP Java SE 7 Programmer II Certification Guide 

Save 40% on OCP Java SE 7 Programmer II Certification Guide with discount codejn15ocp at manning.com.

You

Save 40% on Apache Cordova in Action with discount code jn15acia at manning.com

Ok, that may be just a tiny bit over the top, but most likely there is a bit of truth to it as well. You know how to install Cordova, how to generate native binaries from HTML, and how to make use of fancy device features with plugins. That

By Aditya Y. Bhargava, author of Highlighted text Grokking Algorithms 

Save 40% on Grokking Algorithms with discount code jn15ga at manning.com.

Suppose you are in San Francisco, and you want to go from Twin Peaks to the Golden Gate bridge. You want to get there by bus, in the minimum number of transfers. Here are your options:

 

What

By John Sonmez, Soft Skills: The software developer's life manual 

Save 40% on Soft Skills: The software developer's life manual with discount codejn15ss at manning.com.

Now that you

By Avi Pfeffer

Save 40% on Practical Probabilistic Programming with discount code jn15ppp at manning.com.

 

Imagine you are implementing a highway traffic surveillance system. Your goal is to monitor the flow of vehicles along the highway and predict when traffic jams might occur. You have video cameras placed at strategic points. To accomplish your goal, you will need to do two things. First, you will need to infer the vehicles passing at each point, including identifying when the same vehicle passes two different cameras. Second, you need to predict future vehicles that will arrive on the highway and the traffic jams that result.

Both of these tasks require reasoning about an unknown number of objects. When you are inferring the vehicles currently on the highway, you don

By Chris Birchall, Re-Engineering Legacy Software 

Save 39% on Re-Engineering Legacy Software  withdiscount code relegjn14 at manning.com.

Every developer is occasionally guilty of writing code knowing it

By Avi Pfeffer for Practical Probablistic Programming Save 40% on Practical Probabilistic Programming with code pppjn at manning.com. 

Probabilistic programming is a way to create systems that help us make decisions in the face of uncertainty. Probabilistic reasoning combines our knowledge of a situation with the laws of probability to determine those unobserved factors that are critical to the decision. Until recently, probabilistic reasoning systems have been limited in scope, and have been hard to apply to many real world situations. Probabilistic programming is a new approach that makes probabilistic reasoning systems easier to build and more widely applicable.

To explain probabilistic programming, we

By Kyle Banker, Peter Bakkum, Shaun Verch, Douglas Garrett, and Tim Hawkins MongoDB in Action, Second Edition 

Save 39% on MongoDB in Action, Second Edition discount code jnmongdbat manning.com.

Dedicated text search engines can go beyond indexing just web pages to indexing extremely large databases. Text search engines can provide capabilities such as spelling correction, suggestions as to what you

By Radu Gheorghe and Matthew Lee Hinman, Elasticsearch in Action 

Geospatial data is all about making your search application location aware. For example, to search for events that are close to you, or to find restaurants in a certain area, or to see which park

By Joshua Suereth and Matthew Farwell, SBT in Action 

Save 39% on SBT in Action with discount code sbtjn14 at manning.com.

If you've spent any time working in a team of software developers, you'll have noticed one thing. People make mistakes. Have you ever heard of the old joke about asking 10 economists a question and getting 11 different answers? It's like that when you've got a list of things to do: if you have 10 things to do, you'll make 11 mistakes doing them.

Another thing you'll notice is that people are different. Bob isn't the same person as Ted. When we look at Bob

Filter Blog

By date: