clone and pull all of your company's repositories in a single command

I recently started to work for a new company ( Globality ).

Since we have a very complex setup, it is spread across a large number of
repositories.

During my first days, I wanted to get all of the repositories to my computer.

This might bring some ugly memories, however it is purely for good.

One of my first tasks was to make our docker build faster and I wanted to get a
list of all of our services that are based on python in order to start with
them.

I asked one of my co-workers if he has a script to do that. Sure enough he had
a make file that I tweaked a bit.

Here it is

all:
force:

token=<your-github-token>
user=<your-github-username>
org=<your-github-organization>

auth_opt=-u $(user):$(token)

# note: not all repos are in there!  hitting the page limit of 30.
x.repos.json:  force
                curl -Ks $(auth_opt) https://api.github.com/orgs/$(org)/repos?per_page=100 \
                > $@

clone-all: x.repos.json
          cat x.repos.json \
          | jq -M -r '.[]|.ssh_url' \
          | while read url; do (test -e $$(echo $$url | sed 's|.*/\(.*\)\.git|\1|') || git clone $$url ); done

list-all: x.repos.json
          cat x.repos.json \
          | jq -M -r '.[]|.ssh_url' \
          | while read url; do (echo $$url); done

pull-all:
          for d in ./*; do echo $$d; (cd $$d; git pull); done

Usage

  • <your-github-token>: Get a token on github.
  • <your-github-username>
  • <your-github-org>: Most likely company name for us it’s globality-corp

What I have is a directory called ~/Code/globality, in it’s root I have this
Makefile and I do make clone-all in order to clone all repositories and I can
also do make pull-all to pull all the recent changes across the repositories.

The Creative Network - Live streaming a full stack web app

For the last month or so, I have been live streaming myself coding my day to
day tasks on Twitch, LiveEdu, and Youtube

It’s been a really fun experience but I feel there should be something tying
everything together. One single long-running project can do that.

A little while ago, Stephens and I had an
idea for an app we could pair on as a learning experience for him (We worked
for the same company at the time). We didn’t end up working on it together and
we decided we’re going to do it while live streaming and pairing.

The application

The application is called “The Creative Network” and it as a place where you
can share your creative space for inspiration and for people with the same
hobby to find inspiration.

For example, say you want to share your awesome home office. You can take
pictures, add some description and so on.

So far, nothing special, the really cool feature is that you’ll be able to tag
the images with product links, and people will be able to buy the products
right from your page.

Mockups

Homepage

Product page

These are the homepage and the product page.

Tech stack

Client:

  • React
  • State management: Redux/Mobx
  • npm, webpack, build scripts

Server side:

  • Ruby On Rails?
  • Lambda functions
  • Golang
  • Node.js (GraphQL)

Database:

  • pg
  • Redis
  • ElasticSearch

Mobile:

  • React Native (iOS)

Channels to follow

Schedule

Right now, we are planning on doing this 3 times a week between 9:30pm and 11pm
PST. You can watch it live then OR you can watch the Youtube channel at your
own time. Whatever you decide to do, please participate, ask questions and be
involved.

Configuring multiple reportes for Apache Flink

Having multiple reporters configured for Apache Flink can be really great. You
can send some reports to JMX and graphite.

It’s a really powerful feature and it’s quite easily configured

In your flink-conf.yml you can have this:

metrics.reporters: jmx,graphite

metrics.reporter.jmx.class: org.apache.flink.metrics.jmx.JMXReporter
metrics.reporter.jmx.port: 9020-9040

metrics.reporter.graphite.class: org.apache.flink.metrics.graphite.GraphiteReporter
metrics.reporter.graphite.host: 192.168.1.1
metrics.reporter.graphite.port: 10000

In order for both of these to work you will need to download the jar files and
put them in the lib folder

Devops - getting Apache Flink to work on top of Meos - live stream Feb 20 2017

Today’s project

Streaming time: Feb 20 2017 4:30pm Pacific

Project details

I am introducing a new part into the stack. Mesos.

In order to introduce this, I am using these tech/languages

  • puppet
  • terraform
  • AWS cli
  • mesos

The final result

The final result needs to be that Flink will run on top of mesos in order to
consume messages from Kafka and run tasks.

Multiple API update using Golang - Live Stream Feb 16 2017

Today’s project

Streaming time: Feb 16 2017 9pm Pacific

The problem

Ever since I started streaming to 3 platforms, I faced the problem of updating
the channel title, description and other information.

You have to open the browser, go into each page and update the data, even
though it is essentially the same.

Example screenshots

Youtube

Youtube channel settings

LiveEdu

pic alt

Twitch

pic alt

The solution

I want to have a single place where I update the title and the description
locally.

Let’s say I want to do something like ./stream new, this will create a yaml
file where I will update all of the information. One I update the information,
I do ./stream submit {filename.yml} and that’s it.

The technlology / language

As with any project that is a cli, I choose to go with Golang. I feel it is the
easiest and most friendly for this type of project.

  • Golang
  • Yaml
  • Youtube API
  • Twitch API
  • LiveEdu API

As always, please feel free to ask questions and add your comments during the
stream

I encourage you to ask questions during the stream, I like the interaction and
I love ansering questions.

Satisfying an interface with Golang - Real life example with go-aws SDK

One of the first best practices I started using when coding with Golang is have
the method arity accept the top interface and not the specific class/struct.

This means, that as long as your type satisfies a specific interface, you can
treat your go code as pretty dynamic.

I want to walk though a specific implementation of this I recently had to do
with one of my projects.

The project

  • I needed to clone a database daily from the latest snapshot of production
    database. In order to do that I needed to use the go-aws SDK.

Here’s the code

func (r *RDS) getLatestSnapshotId() (string, error) {
	input := &rds.DescribeDBSnapshotsInput{
		DBInstanceIdentifier: r.Config.DBName,
	}
	output, _ := r.Service.DescribeDBSnapshots(input)

	if len(output.DBSnapshots) == 0 {
		return "", fmt.Errorf("Didn't find any snapshots for %s", r.Config.DBInstanceIdentifier)
	}

  // REST OF CODE HERE
}

output here is of type DescribeDBSnapshotsOutput which contains a slice of
snapshots with a timestamp field called SnapshotCreateTime.

Implementing the sort

Diving into the sort documentation here: https://golang.org/pkg/sort/ we see that
sort.Sort accepts a data Interface which looks like this:

type Interface interface {
        // Len is the number of elements in the collection.
        Len() int
        // Less reports whether the element with
        // index i should sort before the element with index j.
        Less(i, j int) bool
        // Swap swaps the elements with indexes i and j.
        Swap(i, j int)
}

All I need to do is implement this interface on a type and pass it to
sort.Sort, lets dive into this.

I created a file called snapshotslice.go

func NewSnapshotSlice(snapshots []*rds.DBSnapshot) *SnapshotSlice {
	return &SnapshotSlice{
		Snapshots: snapshots,
	}
}

type SnapshotSlice struct {
	Snapshots []*rds.DBSnapshot
}

You can see here that SnapshotSlice holds a reference to the original
rds.DBSnapshots and that’s what we’re going to use.

First method we need to implement is Len()

func (p SnapshotSlice) Len() int {
	return len(p.Snapshots)
}

Pretty straightforward, we just “delegate” the len to the original slice.

The next is Less(), which is a bit more “complicated” but we can explain it
simply by:

The method accepts to indexes on the slice, you need to say whether i is
“bigger” than j remember these since you’ll need to swap the position later.

func (p SnapshotSlice) Less(i, j int) bool {
	t := *p.Snapshots[i].SnapshotCreateTime
	o := *p.Snapshots[j].SnapshotCreateTime
	return t.Before(o)
}

I can use t.Before(o) since the data type for that field is Time, this
saved me a parsing phase if the original library would have returned these as
strings for example.

The last method is Swap which basically swaps one with the other

func (p SnapshotSlice) Swap(i, j int) {
	p.Snapshots[i], p.Snapshots[j] = p.Snapshots[j], p.Snapshots[i]
}

Now, we can do this in our code (later in the getLatestSnapshotId method)

	snapshots := NewSnapshotSlice(output.DBSnapshots)
	sort.Sort(sort.Reverse(snapshots))

Now, I get a sorted slice which I can grab the first item and it’s the latest
snapshot.

Conclusions

Before going into Golang a lot of people have the wrong concept that because
it’s strongly typed you lose a lot of dynamic support. In a way, that’s true
but it’s still not as limiting as you think.

In order to do that, you need to adapt with methods arity that accept the
higher Interface and not the specific type.

When you use a library and it doesn’t do what you want, you can wrap the
libraries objects with your own that implement any method you want and use that
for the rest of your code.

Thank you

Hope you enjoyed reading and this helps you with your code/work.

Puma and webpack proxy aren't playing nicely, here's how you fix it

Today I started a new experiment. I am going to be live-coding some of the many
things I do in my day to day work.

I will blog soon on the reasons for doing this.

During the coding session I encountered a problem where webpack dev proxy is
not working with the puma rack server.

I created a mini go based HTTP server and I verified that this is actually
working for another server first, that’s how I isolated that it’s actually puma
related and not something in webpack or my configuration.

Here’s a sample of the configuration, just for reference:

  devServer: {
    contentBase: './dist',
    hot: true,
    historyApiFallback: true,
    proxy: {
      "/api": {
        target: 'http://localhost:3000'
      }
    }
  }

Turns out with the puma web server this will fail since puma will not bind to
127.0.0.1 and the webpack proxy is likely resolving localhost to 127.0.0.1.

In order to fix this, when you run your rails server, just run it with the -b
option to bind it to 127.0.0.1 like so: rails server -b 127.0.0.1.

That’s it, now you can enjoy working with webpack and puma as your rails-api
rack server.

Scalable (and generic) Mixpanel Tracking for React-Redux applications

If you’ve been following me online, even just for a couple of weeks you’d know that I’m rewriting a major and critical application using React-Redux.

At Gogobot, we A/B test quite a bit and we track behavior in order to make sure what we deliver to our users is top notch quality and not confusing.

One of the most used services around the Gogobot office is Mixpanel which allows you to track custom events with any payload and graph it. You can check with version of your application is behaving better and which does the user like better.

Before React-Redux, we had buttons around the site with .mixpanel class. If one of those button was clicked we would fire the event and listen to it.
Since with React we got rid of jQuery, I wanted to make sure we have a scalable way to do it.

Higher Order Component to the rescue

I’ve written about Higher Order component in the past, feel free to go there and read on if you’re not sure what it means.

Connected Higher Order components with React and Redux

For this, a Higher Order Component really fit. Ideally I’d want something like this.

MixpanelButon component with a trackEvent function. The “child” component (a) in our case will hold data-mixpanel-eventname and data-mixpanel-eventproperties (JSON). Once clicked it will fire the trackEvent function that will fire the actionCreator.

Getting started

Middleware

In order to connect to mixpanel I found this handy mixpanel middleware that does precisely what I want redux-mixpanel-middleware.

All you need to do is dispatch an action with the mixpanel object and the rest is taken care of in the middleware.

Action Creator

import * as actionTypes from '../constants/actionTypes';

export function sendMixpanelEvent(eventName, mixpanelParams = {}) {
  return function(dispatch, getState) {
    let params = mixpanelParams;

    if (typeof(params) === 'string') {
      params = JSON.parse(params);
    }

    dispatch({
      type: actionTypes.MIXPANEL_EVENT,
      meta: {
        mixpanel: {
          event: eventName,
          props: Object.assign({}, params)
        }
      }
    });
  }
}

As you can see it’s pretty simple, I am just dispatching a MIXPANEL_EVENT with the meta object (that’s being taken care of in the middleware). eventName is passed into the creator and the params are either an object OR a JSON string.

I could have also parsed the JSON before I dispatched the action but I figured the API will be cleaner with accepting an object or JSON. Matter of taste here I guess…

The Higher Order Component

import React, { Component } from 'react';
import ReactDOM from 'react-dom';
import { connect } from 'react-redux';
import { sendMixpanelEvent } from '../../actions/mixpanel';
import { bindActionCreators } from 'redux';

function MixpanelButtonWrapper(ComposedButton) {
  class MixpanelButton extends Component {
    render() {
      return(
        <ComposedButton
          { ...this.props }
          { ...this.state }
          trackEvent={ this.props.track } />
      );
    }
  }

  function mapStateToProps(state) {
    return {
    };
  }

  function mapDispatchToProps(dispatch) {
    return {
      track: (e) => {
        const target = e.target;
        const eventName = target.getAttribute("data-mixpanel-eventname");
        const eventAttributes = target.getAttribute("data-mixpanel-properties");

        if (eventName && eventAttributes) {
          dispatch(sendMixpanelEvent(eventName, eventAttributes));
        }
      }
    };
  }
  return connect(mapStateToProps, mapDispatchToProps)(MixpanelButton);
}

export default MixpanelButtonWrapper;

This is the higher order component called MixpanelButton and it’s being wrapped by the MixpanelButtonWrapper function that returns the component.

It’s passing a track prop through the mapDispatchToProps function and that is being passed down into the ComposedButton as trackEvent prop.

The button

import React, { Component } from 'react';
import ReactDOM from 'react-dom';
import * as mixpanelEvents from '../../constants/MixpanelEvents';
import MixpanelButtonWrapper from '../MixpanelButton';

function BookButton(props) {
  const { buttonClassName, price, trackEvent } = props;
  const index = props.index || 1;

  const mixpanelProps = {
    "Booking Partner": price.provider_name,
    "Metasearch Rank": index,
    "Duration of stay": price.total_nights,
    "Dates set": true,
    "Currency Setting": "USD"
  }

  return(
    <a className={ buttonClassName }
      onClick={ (e) => trackEvent(e) }
      target="_blank"
      data-mixpanel-eventname={ mixpanelEvents.HOTEL_TRANSACTION }
      data-mixpanel-properties={ JSON.stringify(mixpanelProps) }
      href={ price.proxy_booking_url }>View deal</a>
  );
}
export default MixpanelButtonWrapper(BookButton);

The button itself is also simple, it’s just an a tag that has an onClick event that fires the trackEvent with the event as a param.

As you can see at the very bottom of the component, I am wrapping the component with MixpanelButtonWrapper.

Now, every button I want to have Mixpanel on, I can do the same thing. Very easy to do and figure out. It’s not as simple as just adding a class like we had previously but it’s simple enough for the use-case.

One more thing…

We don’t really have to pass trackEvent as a prop to the ComposedButton component. Since we spread all the props in {... this.props} the ComposedButton component has access to track as part of it. It’s really a more verbose self documenting way of knowing what the event is.

In action

You can see the button accepting the params as expected. When I click the button the params from the data- attributes are bring pulled in and passed to the action creator and later sent to Mixpanel.

Summary

To sum up. You can have common behaviors in your React applications if you use the code right and connecting those common behavior to Redux is also not a big issue.

I am really having a lot of fun working with the React-Redux combination. Feel free to add your comments and ask questions.

Connected Higher Order components with React and Redux

Recently I’ve started using React and Redux to rebuild one of the major components on the Gogobot website.

In order to understand the problem better, here are some screenshots of one of the components on that page.

Connected Component Screenshot 1

Without really understanding the ins and outs of the component, you can see that the components share many things:

  1. Count label that is shown/hidden based on the number of items I selected from the filter.
  2. Drop down that opens on click of the icon.
  3. Once clicked, it sends an event (action) and requests the data from the server based on the new filter.

Sitting down to design this component I realized that the behavior shared between each of them calls for a higher order component that will define the behavior and wrap all “child” components.

Doing this with pure React is fairly obvious so I’m not gonna go into that, this post is gonna be about doing this with Redux since the higher-order-component is connected to the dispatch, store and state.

Implementation

src/components/Filters/FilterWrapper/index.js

import React, { Component  } from 'react';
import { bindActionCreators } from 'redux';
import { connect } from 'react-redux';

function FilterWrapper(ComposedFilter, filterInfo) {
  class BaseFilter extends Component {
    constructor() {
      super();
		  this.state = {
		  	count: 0
	  	};
      this.onCheckboxChange = this.onCheckboxChange.bind(this);
    }

    onClick(e) {
	
    }

    onCheckboxChange(e) {
	
    }

    render() {
      let countLabel = this.state.count > 0 ?
        <span>{ this.state.count }</span> :
        null;

      return(
        <div className="filterDetailsWrapper">
          <div className="filterTotalCount">
            { countLabel }
          </div>
          <div className="optionsDropDownContainer">
            <ComposedFilter
              {...this.state}
              {...this.props}
              onCheckboxChange={ this.onCheckboxChange }
              />
          </div>
        </div>
      );
    }
  }

  function mapStateToProps(state) {
	// REDACTED
	return {};
  }

  function mapDispatchToProps(dispatch) {
    return {
      ...bindActionCreators(actions, dispatch)
    };
  }
  return connect(mapStateToProps, mapDispatchToProps)(BaseFilter);
}

export default FilterWrapper;

Lets explain what’s going on here…

We’re creating a function that wraps a component and defines some behavior. This is shared behavior to all the wrapped components. In this example we only have the count label for the sake of example but you can obviously extend it to whatever you can.

Let’s take a look at the wrapped component (HotelClass) for this example

src/components/Filters/HotelClass/index.js

import React, { Component } from 'react';
import ReactDOM from 'react-dom';
import BaseFilterWrapper from '../BaseFilterWrapper';

class HotelClass extends Component {
  render() {
    return(
      <div className="hotelClassOptions">
        <ul>
          <li className="optionsdropdown">
            <label className="optionTitle">5</label>
            <input onChange={ this.props.onCheckboxChange } type="checkbox" value="5" />
          </li>
          <li className="optionsdropdown">
            <label className="optionTitle">4</label>
            <input onChange={ this.props.onCheckboxChange } type="checkbox" value="4" />
          </li>
        </ul>
      </div>
    )
  }
}

let filterInfo = {
  name: 'hotel_class',
  class_name: 'hotelClass',
  title: 'Hotel Class'
};

export default BaseFilterWrapper(HotelClass, filterInfo);

As you can see, when the checkbox changes, it’s calling this.props.onCheckboxChange which is coming from the higher order component and in turn will call the behavior there.

The final lines are the “magic” ones, I pass some filterInfo which you can use any way you see fit (or not), and I pass the HotelClass component wrapped in BaseFilterWrapper function.

Now, lets implement the logic to show the count label when we click the checkboxes and hide it if the count is 0

onCheckboxChange(e) {
	let { count } = this.state;
	var { checked, value } = e.target;
	
	if (checked) {
	  count += 1;
	} else {
	  count -= 1;
	}
	
	this.setState({
	  count
	});
}

That’s about it.

Now, you can share similar component logic.

Benefits

The clear benefits of this approach is that if you have a share behavior for a set of components, you only connect the “base” to the store and you manage everything from it. The rest of your components are just “dumb components” and the get injected with the behavior.

The code is much cleaner and there’s a single point of entry for actions/state changes to the system, this simplifies debugging and testing a lot and overall makes the code more readable.

Using webpack to proxy to your local web server to avoid Cross Site errors

You know the pain of this error message I assume…

No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8080' is therefore not allowed access. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled

This is caused by Javascript trying to access a web server not on the same origin. For example http://localhost:8080 trying to access http://localhost:3000.

You can override this by allowing cross origin access, but that’s not usually a good idea.

The best way to go around it is to make your local same as production, your client application and the server are on the same origin.

In production, this is usually handled by Nginx or any other web server, on your local, you can use Webpack in order to achieve this.

So, for example, anything with /api/v5 prefix will be proxied to your web server.

In your devServer segment of your webpack.config.js you can simply add this:

    proxy: {
      '/api/v5*': 'http://localhost:3000'
    }

This basically means that whenever you have a request going to /api/v5/something it will be proxied through your local web server (Rails in my case).

Like so:

    fetch('/api/v5/tribes/index.json').then(res => res.json()).then(function(response) {
    // REDACTED...

You can have multiple proxies (imagine services doing different things) as well.

Hope this helps, enjoy!