scrapy 0.0.2

  • README.md
  • CHANGELOG.md
  • Example
  • Installing
  • Versions
  • new58

scrapy #

pub package

Scrapy, a fast high-level web crawling & scraping framework for dart.

Getting started #

import 'package:scrapy/scrapy.dart';
import 'package:dio/dio.dart';
import 'package:html/parser.dart' show parse;

class Quote extends Item {
  String quote;
  Quote({this.quote});
  @override
  String toString() {
    return "Quote : { quote : $quote }";
  }

  Map<String, dynamic> toJson() => {
        "quote": quote == null ? null : quote,
      };
}

class Quotes<Quote> extends Items {
  @override
  Map<String, dynamic> toJson() {
    return super.toJson();
  }
}

class BlogSpider extends Spider<Quote,Quotes> {
  Stream<String> Parse(Response response) async* {
    var document = parse(response.data.toString());
    var nodes = document.querySelectorAll("div.quote> span.text");

    for (var node in nodes) {
      yield node.innerHtml;
    }
  }

  @override
  Stream<String> Transform(Stream<String> stream) async* {
    await for (String parsed in stream) {
      var transformed = parsed;
      yield transformed.substring(1, parsed.length - 1);
    }
  }

  @override
  Stream<Quote> Save(Stream<String> stream) async* {
    await for (String transformed in stream) {
      Quote quote = Quote(quote: transformed);
      yield quote;
    }
  }
}

main() async {
  BlogSpider spider = BlogSpider();
  spider.name = "myspider";
  spider.start_urls = [
    "http://quotes.toscrape.com/page/7/",
    "http://quotes.toscrape.com/page/8/",
    "http://quotes.toscrape.com/page/9/"
  ];

  Stopwatch stopw = new Stopwatch()..start();
  
  await spider.start_requests();
  await spider.save_result();
  var elapsed = stopw.elapsed;

  print("the program took $elapsed"); //the program took 0:00:00.279733
}

TODOs #

0.0.2 #

  • Badge

0.0.2-1 #

  • Updated README and increas score on pub

0.0.1 #

  • Initial version

example/main.dart

import 'package:scrapy/scrapy.dart';
import 'package:dio/dio.dart';
import 'package:html/parser.dart' show parse;
import 'items.dart';

class BlogSpider extends Spider<Quote, Quotes> {
  Stream<String> Parse(Response response) async* {
    var document = parse(response.data.toString());
    var nodes = document.querySelectorAll("div.quote> span.text");

    for (var node in nodes) {
      yield node.innerHtml;
    }
  }

  @override
  Stream<String> Transform(Stream<String> stream) async* {
    await for (String parsed in stream) {
      var transformed = parsed;
      yield transformed.substring(1, parsed.length - 1);
    }
  }

  @override
  Stream<Quote> Save(Stream<String> stream) async* {
    await for (String transformed in stream) {
      Quote quote = Quote(quote: transformed);
      yield quote;
    }
  }
}

main() async {
  BlogSpider spider = BlogSpider();
  spider.name = "myspider";
  spider.start_urls = [
    "http://quotes.toscrape.com/page/7/",
    "http://quotes.toscrape.com/page/8/",
    "http://quotes.toscrape.com/page/9/"
  ];

  Stopwatch stopw = new Stopwatch()..start();

  await spider.start_requests();
  await spider.save_result();
  var elapsed = stopw.elapsed;

  print("the program took $elapsed");
}

Use this package as a library

1. Depend on it

Add this to your package's pubspec.yaml file:


dependencies:
  scrapy: ^0.0.2

2. Install it

You can install packages from the command line:

with pub:


$ pub get

with Flutter:


$ flutter packages get

Alternatively, your editor might support pub get or flutter packages get. Check the docs for your editor to learn more.

3. Import it

Now in your Dart code, you can use:


import 'package:scrapy/scrapy.dart';
  
Version Uploaded Documentation Archive
0.0.2 Feb 22, 2019 Go to the documentation of scrapy 0.0.2 Download scrapy 0.0.2 archive
0.0.1 Feb 22, 2019 Go to the documentation of scrapy 0.0.1 Download scrapy 0.0.1 archive
0.0.2-1 Feb 22, 2019 Go to the documentation of scrapy 0.0.2-1 Download scrapy 0.0.2-1 archive
Popularity:
Describes how popular the package is relative to other packages. [more]
21
Health:
Code health derived from static analysis. [more]
99
Maintenance:
Reflects how tidy and up-to-date the package is. [more]
90
Overall:
Weighted score of the above. [more]
58
Learn more about scoring.

We analyzed this package on Mar 6, 2019, and provided a score, details, and suggestions below. Analysis was completed with status completed using:

  • Dart: 2.2.0
  • pana: 0.12.14

Platforms

Detected platforms: Flutter, other

Primary library: package:scrapy/scrapy.dart with components: io.

Health issues and suggestions

Document public APIs. (-1 points)

28 out of 28 API elements have no dartdoc comment.Providing good documentation for libraries, classes, functions, and other API elements improves code readability and helps developers find and use your API.

Maintenance suggestions

Package is pre-v0.1 release. (-10 points)

While nothing is inherently wrong with versions of 0.0.*, it might mean that the author is still experimenting with the general direction of the API.

Dependencies

Package Constraint Resolved Available
Direct dependencies
Dart SDK >=2.0.0 <3.0.0
dio ^1.0.14 1.0.17 2.0.18
Transitive dependencies
cookie_jar 0.0.8 1.0.0
Dev dependencies
html ^0.13.3