Bug 397055 - Extern import can't read from stdout
Summary: Extern import can't read from stdout
Status: RESOLVED NOT A BUG
Alias: None
Product: skrooge
Classification: Applications
Component: general (show other bugs)
Version: Trunk from git
Platform: Compiled Sources Linux
: NOR major
Target Milestone: 25.1.0
Assignee: Stephane MANKOWSKI
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-08-01 12:42 UTC by Gerry Agbobada
Modified: 2018-08-06 08:42 UTC (History)
1 user (show)

See Also:
Latest Commit:
Version Fixed In:
Sentry Crash Report:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gerry Agbobada 2018-08-01 12:42:48 UTC
Hello,

I'm on git ( 28b7e22ee (HEAD -> master, origin/master, origin/HEAD) Addition of a new option to check if import has been broken ), and when I tried to import from boobank and boobank_coming (I didn't try this since before 2.14.0 tag to be honest) I get ERR-5 unrecognized date format.

export SKGTRACE=20 ; skrooge ; click on File > Import > Import from extern engines gives in terminal the correct command but apparently the history is only read as chinese / japanese characters (the program segfaults when I activate SKGTRACE, but it always displays this kind of characters) :

##      >virtual SKGError SKGImportPluginBackend::importFile()
##        >virtual SKGError SKGDocument::beginTransaction(const QString&, int, const QDateTime&, bool)
##        Input parameter [name]=[Importation avec Backend]  [nb step]=[3]  [refresh]=[Y]
##        <virtual SKGError SKGDocument::beginTransaction(const QString&, int, const QDateTime&,
 bool) RC=[SUC-0]
##      Execute: PYTHONIOENCODING=utf_8 boobank ls -q -f csv -s id,label,balance 2>/dev/null |grep -v "If --auto-update" | grep -E "id;label;balance|"
##      Read line: __ACCOUNT_1__@__BANK__;__ANON__;11.33
##      Read line: __ACCOUNT_2__@__BANK__;__ANON__ __ANON__;__ANON__
##      Read line: __ACCOUNT_3__@__BANK__;__ANON__ __ANON__;__ANON__
##      Read line: __ACCOUNT_4__@__BANK__;__ANON__ __ANON__ __ANON__;__ANON__
##      Read line: __ACCOUNT_5__@__BANK__;__ANON__ __ANON__ __ANON__;__ANON__
##      Read line: __ACCOUNT_6__@__BANK__;__ANON__ __ANON__;__ANON__
##        >virtual QString SKGDocument::getParameter(const QString&, const QString&) const
##        Input parameter [iName]=[SKG_LAST_WEBOOB_IMPORT_DATE]
##        <virtual QString SKGDocument::getParameter(const QString&, const QString&) const
##        >virtual QString SKGDocument::getParameter(const QString&, const QString&) const
##        Input parameter [iName]=[SKG_LAST_WEBOOB_IMPORT_LIST]
##        <virtual QString SKGDocument::getParameter(const QString&, const QString&) const
##      Mode getoperations
##      llExecute: a="" && b="2000-01-01" && m=$( [[ ${a} > ${b} ]] && echo "$a" || echo "$b" ) && PYTHONIOENCODING=utf_8 boobank -q -f csv history "__ACCOUNT_1__@__BANK__" -s rdate,type,raw,label,amount --condition "rdate>$m OR date>$m OR rdate>$m 00:00:00 OR date>$m 00:00:00" -n 99999  2>/dev/null
##      Execute: a="" && b="2000-01-01" && m=$( [[ ${a} > ${b} ]] && echo "$a" || echo "$b" ) && PYTHONIOENCODING=utf_8 boobank -q -f csv history "__ACCOUNT_2__@__BANK__" -s rdate,type,raw,label,amount --condition "rdate>$m OR date>$m OR rdate>$m 00:00:00 OR date>$m 00:00:00" -n 99999  2>/dev/null
##      llT造。罣罣b="2000-01-01" e%


When I execute the same command in a terminal, I get the expected result (a nicely formatted table with all the transactions).

My locale (in KDE systemsettings5) is fr_FR everywhere except for numbers where I use C
Comment 1 Stephane MANKOWSKI 2018-08-01 13:06:42 UTC
Hi,

The problem is due to date format.
Could you execute the following command for one of your accounts and print here the dates returned?
a="" && b="2000-01-01" && m=$( [[ ${a} > ${b} ]] && echo "$a" || echo "$b" ) && PYTHONIOENCODING=utf_8 boobank -q -f csv history "__ACCOUNT_2__@__BANK__" -s rdate,type,raw,label,amount --condition "rdate>$m OR date>$m OR rdate>$m 00:00:00 OR date>$m 00:00:00" -n 99999  2>/dev/null
Comment 2 Gerry Agbobada 2018-08-01 18:14:51 UTC
In terminal the output is this (only most recent transaction shown)

2018-07-30 08:39:00;TransactionType.WITHDRAWAL;CARTE __CARD_ID__ RETRAIT DAB __BANK_ID__ 30/07 08H39 __ATM_ID__;__ATM_ID__;-20.00
Comment 3 Stephane MANKOWSKI 2018-08-02 06:26:02 UTC
Hi,

Thank you. I understand the issue.
The problem is because the dates are in the format "2018-07-30 08:39:00" instead of "2018-07-30 00:00:00". 
I will do a correction soon.
Regards.
Comment 4 Stephane MANKOWSKI 2018-08-02 20:59:19 UTC
Hi, 
I thought I understood the problem but I didn't. 
I don't reproduce the problem if I import your CSV line. 

Can you confirm me that all dates are in the format 2018-07-30 08:39:00?
Comment 5 Gerry Agbobada 2018-08-03 11:52:39 UTC
Some dates have the time stamp (xx:xx:xx) and others don't, it changes throughout the CSV. Maybe the importer expects only 1 date format for the whole file ?
Comment 6 Gerry Agbobada 2018-08-03 12:03:25 UTC
(In reply to Gerry Agbobada from comment #5)
> Some dates have the time stamp (xx:xx:xx) and others don't, it changes
> throughout the CSV. Maybe the importer expects only 1 date format for the
> whole file ?

Actually, I think I got it. I checked all accounts one by one, and one savings accounts is having bad format (AV Linxea) :
-> % a="" && b="2000-01-01" && m=$( [[ ${a} > ${b} ]] && echo "$a" || echo "$b" ) && PYTHONIOENCODING=utf_8 boobank -q -f csv history "__SAVINGS_ID__@__AV_ID__" -s rdate,type,raw,label,amount --condition "rdate>$m OR date>$m OR rdate>$m 00:00:00 OR date>$m 00:00:00" -n 99999  2>/dev/null
rdate;type;raw;label;amount
NotAvailable;TransactionType.UNKNOWN;FRAIS ANNEE 2018;FRAIS ANNEE 2018;__AMOUNT__
NotAvailable;TransactionType.UNKNOWN;VERSEMENT PROGRAMME;VERSEMENT PROGRAMME;__AMOUNT__
NotAvailable;TransactionType.UNKNOWN;DISTRIBUTION LYXOR UCITS ETF LYXOR UCITS ETF S&P 500 D EUR;DISTRIBUTION LYXOR UCITS ETF LYXOR UCITS ETF S&P 500 D EUR;__AMOUNT__

I think I'll deactivate this import for the time being. Sorry for the confusion with the initial report.

Is there a way to ignore the dates (or import without dates and make Skrooge prevent import operation validation if there's no date ?)
Comment 7 Stephane MANKOWSKI 2018-08-04 10:35:10 UTC
(In reply to Gerry Agbobada from comment #5)
> Some dates have the time stamp (xx:xx:xx) and others don't, it changes
> throughout the CSV. Maybe the importer expects only 1 date format for the
> whole file ?

I did a test, you can mix YYYY-MM-DD HH-mm-ss and YYYY-MM-DD.
Comment 8 Stephane MANKOWSKI 2018-08-04 10:41:01 UTC
(In reply to Gerry Agbobada from comment #6)
> Is there a way to ignore the dates (or import without dates and make Skrooge
> prevent import operation validation if there's no date ?)

The boobank plugin launches the following command:
X-SKROOGE-getoperations=a="%parameter2" && b="%4" && m=$( [[ ${a} > ${b} ]] && echo "$a" || echo "$b" ) && PYTHONIOENCODING=utf_8 boobank -q -f csv history "%1" -s rdate,type,raw,label,amount --condition "rdate>$m OR date>$m OR rdate>$m 00:00:00 OR date>$m 00:00:00" -n 99999 %parameter3 2>/dev/null

As you can see there is %parameter3?

So, in settings, if you set the configuration like this, this should work:
weboob(,,grep -v NotAvailable)
Comment 9 Gerry Agbobada 2018-08-06 08:42:14 UTC
I just tested it (sorry for the delay), it works perfectly. Thanks !