Austin Story

Ruby, Rails and Javascript Blog

Powered by Genesis

Local Copy of Heroku

August 29, 2016 By Austin Story Leave a Comment

Magic command to get a copy of heroku database locally.

local_database = name of your postgres database, will normally be app_name_development
remote_heroku_url = the subdomain before herokuapp.com at heroku

heroku pg:pull DATABASE_URL local_database –app remote_heroku_url

copy one to another

heroku pg:copy production-url::DATABASE_URL DATABASE_URL –app staging-url

Filed Under: Uncategorized

Redirecting To Dynamic Page Using Javascript

January 14, 2015 By Austin Story Leave a Comment

So I recently had a consulting opportunity where the client needed to be sent to a dynamic page after a controller action in rails.  That is easy in the normal rails context, however this all had to be done in a javascript content due to it being a javascript based subscription form for sign up.  I ended up with a nice little parlor trick that is attached below.

A little background, this client used recurly to do recurring billing and recurly uses a javascript system to securely handle creditcard data.  On submit, javascript sends recurly the credit card data and they send you back a token representing the data in their system or you get back errors that you can display to the browser.

Once you have the token there are a couple ways you can approach charging it to add a subscription.  The way that I do it is by setting up a subscription/order controller in rails, that then takes care of creating the subscription through an Ajax request. That allows us to catch any errors (i.e. not enough funds) without forcing the user to re-enter any non-credit card data on the site.

Normally after a successful subscription, you will let the form continue to a subscription/registrations controller and that will take care of the setup.

In this app, there were multiple paths that the user could go based on what they order and their progress in the system. So i needed to redirect them dynamically. The way that i did that is on success instead of allowing the form to continue using javascript, i called the function below.

[ruby]
function redirectOrder(url){
var plan = $(‘#recurly_plan_code’).val();
var form = $(‘<form action="’ + url + ‘" method="get">’ +
‘<input type="text" name="recurly_plan_code" value="’ + plan + ‘" />’ +
‘</form>’);
$(‘body’).append(form);
form.submit();
}
[/ruby]

Filed Under: Uncategorized

Browser Scraping and Data Mining Part 1

October 5, 2013 By Austin Story 2 Comments

I am getting a higher number of requests for collecting data off websites on the internet.  I wanted to share how I approach data mining and browser scraping.  Keep in mind that all websites have their own terms of use and rights to the content.  Make sure you have the right to scrape data before you do.

Tools

I am a ruby on rails programmer so my natural flow is to use active record and ruby to solve just about all my programs.  So here is what I use

  1. Ruby 2.0.0+ and Rails
  2. Linux Ubuntu 12.04
  3. Chrome Dev Tools
  4. Nokogiri gem- Parse HTML into a neat easy data structure
  5. Watir Webdriver gem – Browser Scraping
  6. StreetAddress gem – Parses addresses using an old perl algorithm.

Important extra gems and versions that work together well as of October 2013

[ruby]
gem ‘nokogiri’, ‘1.5.6’
gem ‘watir-webdriver’, ‘0.6.2’
gem ‘StreetAddress’, ‘1.0.3’, :require => "street_address"

[/ruby]

Approach

  1. Presentation – Determine how the website is giving the data (table, iframe, etc)
  2. Navigation – Determine how to cycle through the website to gather more pages of data

During Part 2 I will be focusing on the presentation part of this and how to write some ruby code to grab data from websites.

Filed Under: Programming, Ruby on Rails, Uncategorized Tagged With: Browser Scraping, Data Mining, Nokogiri, ruby, ruby on Rails, Watir-webdriver

  • « Previous Page
  • 1
  • 2

Categories

  • AngularJS
  • Books
  • Devise
  • Elasticsearch
  • ES6
  • Information Security
  • Integrations
  • Javascript
  • Linux
  • Minitest
  • PhoneGap
  • Programming
  • React
  • Redux
  • Ruby
  • Ruby on Rails
  • Stripe
  • Testing
  • Theory
  • TypeScript
  • Uncategorized
  • Vue
  • Webpack