Cool QL Cool


CoolQLCool (CQC) is an open source Graph QL server that allows you to turn websites into a Graph QL api. It's pretty tubular (I'm incredibly biased tho).

You can play around with it in graphiql. Or take a gander at the source on Github.

Example

Features

In order of how radical I think they are.

  • Waits for JS to do stuff if you specify.

    By default CQC will load just the HTML like a normal AJAX request, but if you pass the `wait` or `waitForSelector` arguments in your initial `site` search, CQC will wait a designated time, or for a CSS selector on the page to show up.

  • Allows you to query multiple sites at once

    In past renditions of web scrapers they've only been able to scrape a single site at once (e.g. Jam API). With CQC you can query multiple sites in a single request by aliasing each `site` call.

  • Easier to add more fields in the future.

    With past scrapers it was incredibly difficult to add new fields to query elements by or to grab off of elements. With CQC it will be incredibly easy to have contributors add new fields and functionality.

Possible Features

Some possible features that I'd like to see CQC have some day

  • Provide steps to do on a page when it loads
  • Recursive fields, for example for pagination on Hacker News to go to a certain number of pages and query each one for the same fields.
  • Field to return the output of javascript

Curl

curl -X GET -H "Content-Type: application/json" -d '{"query": "{ site(url: \"https://news.ycombinator.com\") { titles: selectAll(elem: \"tr.athing\") { id: attr(name: \"id\") numberOfLinks: count(elem: \".storylink\") link: selectAll(elem: \".storylink\") { text href class classList } } } }"}' https://coolql.cool/graphql

Javascript

var xhr = new XMLHttpRequest();
xhr.responseType = 'json';
xhr.open('POST', 'https://coolql.cool/graphql');
xhr.setRequestHeader('Content-Type', 'application/json');
xhr.setRequestHeader('Accept', 'application/json');
xhr.onload = function () { console.log('data returned:', xhr.response);
}
xhr.send(JSON.stringify({ query: `{ site(url: "https://news.ycombinator.com") { titles: selectAll(elem: "tr.athing") { id: attr(name: "id") numberOfLinks: count(elem: ".storylink") link: selectAll(elem: ".storylink") { text href class classList } } }
}` }));

Ruby

require 'net/http'
require 'uri'
require 'json' uri = URI.parse("https://coolql.cool/graphql") header = {'Content-Type': 'application/json'}
query = { query: '{ site(url: "https://news.ycombinator.com") { titles: selectAll(elem: "tr.athing") { id: attr(name: "id") numberOfLinks: count(elem: ".storylink") link: selectAll(elem: ".storylink") { text href class classList } } }
}' } http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri, header)
request.body = query.to_json response = http.request(request)
Made by Gavin, deployed on Now