Week 36

Adventures with Docker I: Virtual Machine creation

Over the last week, this blog has been migrated from dedicated hardware to the Digital Ocean cloud hopefully without readers noticing too much difference.   Special mention needs to be made here of Ian @ Projany Limited who have been kindly hosting the site entirely for free from the outset. Thanks very much for undercutting every cloud provider out there! Moving to Digital Ocean should involve a PAYG cost of around $10/month on current projections.  The initial migration of WordPress was managed using a recipe provided here involving these steps:

  • Back Up all files on old website
  • Export the old WordPress database
  • Create a WordPress database on new host
  • Edit the corresponding wp-config.php file
  • Import old database to new host
  • Upload WordPress files to new host
  • Setup up new DNS details and wait for propagation

A key benefit of moving to the Digital Ocean platform is that it opens up the possibility of automated site management using Docker, a tool that has had a meteoric rise in popularity over the last year amongst devops practitioners.  In order to get started you need to first create a personal access token using the Digital Ocean web admin GUI:

digitalOceanAdmin

Then you need to install a co-ordination tool called Docker Machine to manage the interaction between the Docker command line client and a host platform for running docker on which can be either in the cloud or in a local environment.  Instructions on how to do that are available here. In my case I installed Docker Machine on a standard-issue Windows 7.1 laptop.

Docker Machine works with different infrastructure options through the means of virtual “drivers”.  There are drivers for AWS, Digital Ocean, Azure and VirtualBox.   It’s a very powerful abstraction mechanism that makes it straightforward to try out different cloud provider platforms.   It’s helpful do understand a little more about Docker Machine’s architecture and how it works under the hood with containers and images.  Two useful starting points are docker’s own description and this resource.   Installing Docker Machine gives you all the equipment you need to fully operate docker end to end via a command line tool called docker-machine:

Docker Architecture Diagram

One thing you may want to do post-installation on Windows is configure Docker and Docker Machine git access from the cmd command line thus:

C:\> set PATH=%PATH%;"c:\Program Files (x86)\Git\bin"

At this point creating and launching a new cloud-based Digital Ocean Virtual Machine (VM) called ‘dodocker‘ from within the standard Windows cmd command line takes less than a minute:

C:\> docker-machine create \
--driver digitalocean \
--digitalocean-access-token <access-token> \
 dodocker

C:\> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
dodocker digitalocean Running tcp://104.131.84.90:2376

Once you have this VM running you can interact with it using the standard Docker command line client simply called docker.   This requires configuring some additional environment variables in your Windows cmd session as follows:

C:\> docker-machine env --shell cmd dodocker
set DOCKER_TLS_VERIFY=1
set DOCKER_HOST=tcp://104.131.84.90:2376
set DOCKER_CERT_PATH=\machines\dodocker
set DOCKER_MACHINE_NAME=dodocker
# Run this command to configure your shell:
# copy and paste the above values into your command prompt

With the Docker client thus configured, you can run a command in a new container on the Digital Ocean virtual machine.  Here we try out a vanilla nginx image which brings up an nginx web server on the target VM on external port 8000:

C:\> docker run -d -p 8000:80 nginx
5cb1893cedf093705d2a4af3b07ba4a97f3a79f501bfb2574c45fc3746e0d27b

C:\> docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
73c91ccca3a9 nginx "nginx -g 'daemon off" About a minute ago Up About a minute 443/tcp, 0.0.0.0:8000->80/tcp

C:\> docker ip dodocker
104.131.84.90

If you now direct your browser to http://104.131.84.90:8000/ you should see a “Welcome to nginx!” page.

Using the command line to orchestrate all this cloud interaction on Windows feels impressive.   So does the cornucopia of extant images to play with.  A couple of months ago, I outlined a convoluted recipe for pulling together a deep dreaming inceptionism setup. I needn’t have bothered.  Many people have been there, done it and dockerized it already:

C:\> docker search deepdream
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
ryankennedyio/deepdream A container to run the deep dream python n... 12
mjibson/deepdream Google Deepdream visualization 7
...

The basic workflow with Docker consists of starting collections of containers on a VM.  In that way, it’s possible to run potentially a dozen or so small web sites within corresponding nginx containers running on a single VM.  Posit: if you charged each owner $1/month you’d still be making money…  Before finishing off, it’s important to be a responsible cloud citizen and switch off (and ultimately remove) an instance you’re no longer using:

C:\> docker kill 73c91ccca3a9
73c91ccca3a9

C:\> docker-machine kill dodocker

C:\> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
dodocker digitalocean Stopped tcp://104.131.84.90:2376

C:\> docker-machine rm dodocker
Successfully removed dodocker

There’s a huge lot more you can do with Docker.   The next step for me is to build on this starting point to automate the backup and restore of the entire WordPress site.  @charltones recently outlined a recipe to broadly do just that using a tool called docker compose.

Manufacturers and Devices

What happens when almost everyone on earth has a pocket supercomputer connected to the internet? It’s not a subset of the internet – it IS the internet.

But HTC was neither stagnant nor incompetent.  It regularly introduced novel features, many of which were smart and useful. It created consistently good smartphones. It seemingly did enough to, if not lead the pack, at least stay in its general vicinity. … In fact, what did HTC in wasn’t HTC. It was a smartphone market that simply doesn’t have room for more than a handful of players, one in which being different no longer counts for much.

Apple's ASP has risen in the past year

“More than ever, the smartphone business is turning into one where only two companies make money from handsets – and increasingly, only one gets the top-end business. For the rest, they need something new to come along that they can get into and make some profit. Android Wear smartwatches don’t seem to be setting the world alight; will VR be the next big thing? Or IoT? Or self-driving cars? At this point, anyone that isn’t Apple or Samsung seems to be praying for a miracle.”

“these companies need to look toward innovations that can actually make the human experience more enjoyable. The first smartphones did this by opening up the internet in all of its glory; the first portable music players freed us from having to carry around physical media; wearables promise to make us healthier; and the Internet of Things paves the way for a fully automated home. Genuine usefulness is what these companies need to focus on because simply adding another layer of gloss on what already exists is simply not good enough.”

Google and Android

  • That new Google logo received more coverage and considerable positive reaction online particularly in respect of the claim that the change resulted in the amount of data required for the logo shrinking from 14kbytes for the old serif format to just 305 bytes.  This insightful post on Quora includes the source for a sub-300 byte svg “constructed almost entirely from circles and rectangles” that very closely approximates the new logo.  Here’s what an online renderer comes up with:

google

Asia

Cloud and DevOps

  • Great post courtesy of SpaceApe Games on the devops hierarchy of needs which provides a glimpse at their journey to a devops way of working.  It also posits somewhat cheekily that the top of the triangle is attending and speaking at conferences.  Actually this is more plausible – a view of devops as akin to gardening with no fixed end to aim at only continuous improvement:

More likely is that there is no next level, just a continuous re-hashing of the foundations beneath us as new technologies and challenges enter the fray.

 

  • Serving to underline that message, this Reddit devops thread explains why there isn’t really a “one size fits all” product for the devops space.  Each situation and context is different.  What unites devops practitioners is a shared collection of process, tools and above all shared culture.  It’s really more a movement than any specific technology or product:

Labeling something “DevOps” doesn’t really make a lot of sense, and your use of the word will put you at odds with a significant number of people. Continuous Delivery is OK, I guess, but you better be damned ready to tell me how your product meets the criteria and why Jez Humble and Dave Farley would sing your praises.

As back-end IT is increasingly commoditised and delivered as a utility, as employees increasingly use mobile devices and ubiquitous network access the requirement for the CIO’s huge and costly apparatus diminishes.  DevOps and continuous delivery are similarly rendering the old structures obsolete.

Big Data and Privacy

  • Fascinating and rather disturbing post on what happened next when Australian reporter Will Ockenden exercised his rights to pull a year’s worth of his smartphone network data courtesy of Telstra and then provided it to the general public for data mining purposes.  Among other highly sensitive personal information it revealed his daily location “heat map” and contact graph from an apparently fairly innocuous (at first sight) dataset:

The dataset contained a year’s worth of outgoing call and SMS records, and six months of his ‘data sessions’, which are the records kept every time his phone connected to the internet over the mobile network.

Work hours: Heatmap of Will's approximate location between 11am and 3pm

“Addressing issues of the politics of algorithms, normative classifications and algorithmic governmentality “Algorithmic Regimes and Generative Strategies” wants to mobilize the critical perspectives of researchers, artists and activists, to open the field for a wider and more diverse debate.”

Product Analytics

A data native is someone who expects their world to not just be digital, but to be smart and to adjust immediately to their taste and habits,” she says. “For example, a magazine should not only be digital and interactive — it should be personalized. It should tell you what you need to know based on your interests, location, preferences. The expectations have shifted.

Wearables, the Internet of Things and Hardware

This is the smartwatch that luxury brands should be making

GW IoT_hierarchyOfNeeds pyramid

Technics quartz synthesizer direct-drive turntables, 1999.

Artificial Intelligence and Robotics

  • Provocative and thought-provoking Kernel post on the impact ASI could have on organised religion.  A range of opinions are provided from those that think religion will adapt and evolve as it always has to those that believe artificially superintelligence is fundamentally incompatible with religiosity and will result in a crisis of meaning:

There is no specific religious suggestion that we’ll face a technological singularity. In fact, ancient scriptures from various religions say virtually nothing about science and technology, and what they do say about them is usually wrong (the Earth doesn’t move, is at the center of the solar system, is 6,000 years old).

What makes Brett’s brain tick is a combination of two technologies that have each become fundamental to the AI field: deep learning and reinforcement learning. Deep learning helps the robot perceive the world and its mechanical limbs using a technology called a neural network. Reinforcement learning trains the robot to improve its approach to tasks through repeated attempts.

  • The post is cutely entitled “pre-school for robots” and comes with standard issue robot playing with lego shots. Human analogies stop there though, as the author points out:

“it’s important to remember that A.I. is, and will likely remain, deeply alien”

  • As a counterpoint to the surfeit of robotic determinism and automation dystopia, John Markoff the author of “Machines of Loving Grace” explains why “humanity is essential to the future of artificial intelligence.”   And this interesting Nautilus post explains why smart machines rather than being in binary opposition to humanity seeking our destruction will instead “take us with them” resulting in a complex spectrum of post-human enhancements including genomic tinkering to produce “super humans”:

The feedback loop between algorithms and genomes will result in a rich and complex world, with myriad types of intelligences at play: the ordinary human (rapidly losing the ability to comprehend what is going on around them); the enhanced human (the driver of change over the next 100 years, but perhaps eventually surpassed); and all around them vast machine intellects, some alien (evolved completely in silico) and some strangely familiar (hybrids)

  • In a similar spirit of human-machine collaboration, IEEE outline Toyota plans for a “major push into AI and robotics“.  The focus is not on autonomous vehicles per se but rather on using AI to make more reliable cars:

The initial focus will be on accelerating the development of AI with applications to smarter and safer vehicles, as well as robots that can make our lives better at home, especially as we age.

Management

“For most people, this doesn’t come naturally. Evolution and the survival imperative have attuned human beings to pay careful attention to all things negative; in addition, people quickly become so accustomed to positives in their environments that they hardly notice them. But a simple intervention can help overcome both tendencies.”

There is something called the negativity bias, which says basically, we generally spend more time thinking about negative things than we spend on positive.  So, even if you have appreciated folks you work with it is likely you can appreciate them even more.  It can’t hurt to appreciate more even if you are doing it now.

Startups

  • Vanity Fair outlines the evidence suggesting we’re on the verge of another tech bubble about to burst driven by stratospheric VC driven funding, FOMO (fear of missing out) and a need to “run with the unicorns” in order to hire the best engineers.  Logic suggests it must end somewhere in what the author calls a “kaboom!”.   However, if and when it does, the least likely victims will be VCs who are well-protected against any imminent downside:

The problem with being a unicorn, indeed, is that there aren’t many exit strategies. Either you can go public, which is inadvisable without a lot of revenue, or you can sell, which is difficult given the paucity of companies that can afford to make such an offer. So, for many, the choice becomes fairly simple. You continue to raise more and more money, or you die.  KABOOM!

  • Tech journalist Biz Carson reviewed 146 recent pitch decks and came up with some conclusions that are worthy of enumeration:
    • Security is the new hot ticket.
    • Hardware startups are growing.
    • How we hire and recruit employees is about to change dramatically.
    • Social startups are dead.
    • Money is falling from trees, and even small businesses want a part of it.
    • Everyone is now an “X for Y.”
    • Startups are looking outside Silicon Valley — way outside.
    • Startups are going niche, in both their product and their market.

Science

  • The possibility that we live in a multiverse of universes is one which arises as a consequence of quantum theory and string theory.  This article outlines an approach for how it can be scientifically tested:

The universes predicted by string theory and inflation live in the same physical space (unlike the many universes of quantum mechanics which live in a mathematical space), they can overlap or collide. Indeed, they inevitably must collide, leaving possible signatures in the cosmic sky which we can try to search for.

EGS8p7 is nothing but a dot to space telescopes

Work Antipatterns

A guy who wrote a share market algorithm with 67% accuracy statistically failed in Maths aptitude test in on-campus placement four times in a row.

“By giving employees more of a say in how they do their jobs.  By making sure we offer them opportunities to learn and grow. And by encouraging them to suggest improvements to the production process and listening to what they say.”

The economic explanation is that long-term employment has essentially vanished: Instead of spending our careers at one organization, we expect to jump ship every few years. Since we don’t plan to stick around, we don’t invest in the same way. We view co-workers as transitory ties, greeting them with arms-length civility while reserving real camaraderie for outside work. At best, as the movie “Fight Club” termed our conversation partners on airplanes, colleagues become “single-serving friends.”

Culture and Society

  • Excoriating WashPo post on Syria asks why some Arab nations which are physically and culturally closer to would-be migrants aren’t doing more to support them leaving many to risk all on a perilous journey to Europe.

Many people are afraid to be honest about what they feel and don’t want to do because they believe telling the truth will disappoint and deeply hurt others.

People who are afraid of being treated unfairly will unintentionally treat others unfairly in an attempt to ensure that no one is taking advantage of them.

The Man in the Machine’s most affecting and illuminating moment occurs during an interview with Bob Belleville, a former Apple engineer from the original Mac team. He describes a fairly demanding work environment, where everyone slaved around the clock to get the Mac up and running. That job eventually cost Belleville his marriage and his relationship with his children. But he also had a special connection with Jobs — they traveled to Japan together several times and worked closely for years. Belleville’s dueling mixture of love and underlying contempt for his former boss are plainly visible when, in tears, he reads a eulogy he wrote for Jobs on camera.