Introduce Pytest, Tox, Coverage #19

dbrgn · 2014-09-29T21:35:38Z

Fixes #13.

Still work in progress. Qemu on Travis is very slow :/

The question is whether we actually need Qemu with a real server, or if we can go with mocking one. Then we could get rid of the Qemu setup.

ghickman · 2014-09-29T22:51:36Z

setup.py

+from setuptools import setup, find_packages
+from minecraft import meta
+
+readme = io.open('README.rst', mode='r', encoding='utf8').read()


This can use with open().

The default open is not a good idea because it does not always handle encoding correctly, see strongswan/swidGenerator#35. If the system has a terminal which is not set to UTF-8, it could cause problems.

Using a context manager is a good idea though.

ghickman · 2014-09-29T22:56:39Z

@dbrgn – This is a good start but I don't think we need to target 3.4 as Raspbian doesn't ship it yet. Thanks for making the change to Tox!

doismellburning · 2014-09-30T06:42:16Z

That said, I think we still want to work with 3.4 (for when Raspbian does ship it), but we need it to work for 3.2 (Wheezy stable)

dbrgn · 2014-09-30T07:43:57Z

That said, I think we still want to work with 3.4 (for when Raspbian does ship it), but we need it to work for 3.2 (Wheezy stable)

Exactly, 3.2 should be the main goal right now, but eventually the library will be used under Python 3.4.

I'd suggest we keep the versions 3.3 and 3.4 in tox, but remove them from Travis, what do you think? (Or we could flag them as allowed failures.)

ghickman · 2014-09-30T08:31:32Z

@dbrgn @doismellburning – No I think those are fair points, lets keep both versions in ready for when they're available.

- Tests are run using pytest - Test environments are set up using tox - Right now, Travis-CI is set up using a Qemu installation, in which tox is run - Coverage is measured - Coding guidelines are enforced

dbrgn · 2014-09-30T09:26:45Z

OK, feedback has been integrated and the branch has been rebased.

Right now there's a build matrix on Travis with Python 2.7 and 3.2. There's also a build that does coverage testing, with an allowed failure. In there we could add a tool like Coveralls.

The tests pass on Python 2.7 and fail on all Python 3 versions.

In the "cov" build you can see an ASCII coverage report:

doismellburning · 2014-09-30T09:35:01Z

That Py3 failure looks...fun :s

pozorvlak · 2014-09-30T09:35:52Z

Yay for working CI builds! OTOH, it's a bit disappointing that it fails on
the Python dialect we're supposed to be targeting...

As for the QEMU thing: that was me following instructions from a random
blog post in a code sprint. We're currently doing a ridiculous number of
levels of virtualisation - QEMU within an ARM VM which is almost certainly
being simulated on x86 hardware - and as anyone who's seen Inception
knows, running a VM within a VM within a VM makes everything
veeeerrrrrryyyyy ssssslllllllooooooowwwwwww. If there's a faster way of
testing in our expected deployment environment, that would be awesome;
however, I'm reluctant to go a test setup that doesn't attempt to run at
least a few tests against the actual binary blob.

On Tue, Sep 30, 2014 at 10:26 AM, Danilo Bargen [email protected]
wrote:

OK, feedback has been integrated and the branch has been rebased.

Right now there's a build matrix on Travis with Python 2.7 and 3.2.
There's also a build that does coverage testing, with an allowed failure.
In there we could add a tool like Coveralls.

[image: img]
https://camo.githubusercontent.com/6b89598410c052f0366e3ac665c5713f50acc2d9/687474703a2f2f746d702e646272676e2e63682f73637265656e73686f74732f32303134303933305f3131323330332e706e67

The tests pass https://travis-ci.org/py3minepi/py3minepi/jobs/36648632
on Python 2.7 and fail
https://travis-ci.org/py3minepi/py3minepi/jobs/36648633 on all Python 3
versions.

In the "cov" build you can see an ASCII coverage report:

[image: img]
https://camo.githubusercontent.com/fe6e68de1f74ac81bb3ae7f6288e35d5cae56c42/687474703a2f2f746d702e646272676e2e63682f73637265656e73686f74732f32303134303933305f3131323532332e706e67

—
Reply to this email directly or view it on GitHub
py3minepi/py3minepi#19 (comment).

doismellburning · 2014-09-30T09:44:35Z

Yeah, it's slow and tedious but I definitely like the idea of Actual Realistic Testing - not sure I have any bright ideas about good ways to speed it though...

(DIY Jenkins testing against pre-built emulator images? That way at least we're pre-loading the work from most of https://github.com/py3minepi/py3minepi/blob/master/.travis-ci.sh, otoh that's a tedious maintenance requirement - I think I'd rather just have slower tests...)

doismellburning · 2014-09-30T09:51:54Z

Anyway this looks good. Can I confirm that this is a net improvement, i.e. some tests now pass whereas none did previously?

pozorvlak · 2014-09-30T09:57:56Z

+1 on it being a net improvement, and +1 for the overall idea of the change.

ghickman · 2014-09-30T10:02:14Z

I'm in agreement with @pozorvlak, it's a net improvement and adds some of the items we discussed in #14.

dbrgn · 2014-09-30T11:29:40Z

That Py3 failure looks...fun :s

Probably some issue with identity comparison. But that test doesn't really matter anyways. Most of the code is still untested.

If there's a faster way of testing in our expected deployment environment, that would be awesome

Yes, mocking. But mocking is always based on assumptions. We can just test what gets sent to Minecraft, not what Minecraft does with it.

however, I'm reluctant to go a test setup that doesn't attempt to run at least a few tests against the actual binary blob.

While that sounds good, how do we actually write the assertions? There's no way to test whether a set_block command actually created a block. And Minecraft does not return any output on errors. So basically there's no way to test any assertions using the Minecraft binary.

pozorvlak · 2014-09-30T12:40:55Z

Why can't we send getBlockWithData calls to check if the block now has
the expected type?

On Tue, Sep 30, 2014 at 12:29 PM, Danilo Bargen [email protected]
wrote:

That Py3 failure looks...fun :s

Probably some issue with identity comparison. But that test doesn't really
matter anyways. Most of the code is still untested.

If there's a faster way of testing in our expected deployment environment,
that would be awesome

Yes, mocking. But mocking is always based on assumptions. We can just test
what gets sent to Minecraft, not what Minecraft does with it.

however, I'm reluctant to go a test setup that doesn't attempt to run at
least a few tests against the actual binary blob.

While that sounds good, how do we actually write the assertions? There's
no way to test whether a set_block command actually created a block. And
Minecraft does not return any output on errors. So basically there's no way
to test any assertions using the Minecraft binary.

—
Reply to this email directly or view it on GitHub
py3minepi/py3minepi#19 (comment).

dbrgn · 2014-09-30T13:17:21Z

Oh, true, we can do that... Sorry for forgetting :) It's an integration though, not a unit test. For unit tests, we should use mocking. It's faster anyways.

In that case I'll merge this.

Introduce Pytest, Tox, Coverage

doismellburning · 2014-09-30T13:18:37Z

Please never again merge your own pull request, thanks

dbrgn · 2014-09-30T13:20:42Z

@doismellburning I thought the three confirming comments above were a "good to merge". But in that case I'll leave it to others in the future :)

pozorvlak · 2014-09-30T16:19:41Z

@dbrgn: absolutely, anything that touches the Minecraft binary is an
integration test. But we should do integration tests in addition to our
unit tests, and the place to do so is in CI :-)

On Tue, Sep 30, 2014 at 2:20 PM, Danilo Bargen [email protected]
wrote:

@doismellburning https://github.com/doismellburning I thought the three
confirming comments above were a "good to merge". But in that case I'll
leave it to others in the future :)

—
Reply to this email directly or view it on GitHub
py3minepi/py3minepi#19 (comment).

Fixed PEP8 issues

c610948

ghickman reviewed Sep 29, 2014
View reviewed changes

Configured first testing setup

3af6501

- Tests are run using pytest - Test environments are set up using tox - Right now, Travis-CI is set up using a Qemu installation, in which tox is run - Coverage is measured - Coding guidelines are enforced

dbrgn force-pushed the pytest branch from de7e1e6 to 3af6501 Compare September 30, 2014 09:01

Added coveralls configuration

4491667

dbrgn added a commit that referenced this pull request Sep 30, 2014

Merge pull request #19 from py3minepi/pytest

456702b

Introduce Pytest, Tox, Coverage

dbrgn merged commit 456702b into master Sep 30, 2014

dbrgn deleted the pytest branch September 30, 2014 13:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Pytest, Tox, Coverage #19

Introduce Pytest, Tox, Coverage #19

dbrgn commented Sep 29, 2014

ghickman Sep 29, 2014

dbrgn Sep 30, 2014

ghickman commented Sep 29, 2014

doismellburning commented Sep 30, 2014

dbrgn commented Sep 30, 2014

ghickman commented Sep 30, 2014

dbrgn commented Sep 30, 2014

doismellburning commented Sep 30, 2014

pozorvlak commented Sep 30, 2014

doismellburning commented Sep 30, 2014

doismellburning commented Sep 30, 2014

pozorvlak commented Sep 30, 2014

ghickman commented Sep 30, 2014

dbrgn commented Sep 30, 2014

pozorvlak commented Sep 30, 2014

dbrgn commented Sep 30, 2014

doismellburning commented Sep 30, 2014

dbrgn commented Sep 30, 2014

pozorvlak commented Sep 30, 2014

Introduce Pytest, Tox, Coverage #19

Introduce Pytest, Tox, Coverage #19

Conversation

dbrgn commented Sep 29, 2014

ghickman Sep 29, 2014

Choose a reason for hiding this comment

dbrgn Sep 30, 2014

Choose a reason for hiding this comment

ghickman commented Sep 29, 2014

doismellburning commented Sep 30, 2014

dbrgn commented Sep 30, 2014

ghickman commented Sep 30, 2014

dbrgn commented Sep 30, 2014

doismellburning commented Sep 30, 2014

pozorvlak commented Sep 30, 2014

doismellburning commented Sep 30, 2014

doismellburning commented Sep 30, 2014

pozorvlak commented Sep 30, 2014

ghickman commented Sep 30, 2014

dbrgn commented Sep 30, 2014

pozorvlak commented Sep 30, 2014

dbrgn commented Sep 30, 2014

doismellburning commented Sep 30, 2014

dbrgn commented Sep 30, 2014

pozorvlak commented Sep 30, 2014