Agile business intelligence

Scrum methodology seems to work quite well in projects where there's a well defined scope, everybody knows quite well how a solution should and will be implemented and the timeframe of the work could be predicted with a quite good accuracy. This usually means that scrum methodology is great on low level programming or pure development tasks, also in electronics, robotics, etc.
Unfortunately with business intelligence and data warehousing projects this doesn't always seem to be the case as this is a more particular domain.

Common problems and issues of implementing scrum methodology in business intelligence projects

  • Trying to squeeze tasks scope into one sprint jira tickets no matter what. In BI there are tasks which should not be split, they're just simply big and unpredictable. Forcing them to split into smaller subtasks very often creates confusion and blurry visibility on the management side. And the developers get confused as well.
  • Planning and assigning story points to a task. There's usually no way to predict the amount of workload for a task without actually starting working on it. Practice shows that usually the tasks are underestimated and then a task jumps from one sprint to another for a few weeks or months.
  • Tasks are not well defined. The product owner's responsibility is to make each sprint task as clear as possible but in real life projects the definition of a task is something like 'we need a report based on data x coming from system y and z'. And that's about it.
  • While in traditional programming scrum process working piece of software is the measure of progress, in BI project it's usually not that clear. In most cases there's ETL, Data warehousing and then Reporting part involved which is sort of a backbone building activity which might take weeks or months and only then the results are visible.
  • The daily scrum team meeting (standup). It's really good if the whole team is working on the same task or topic. This gives opportunity to talk things through, share ideas and identify problems. Also it's a great way to keep in touch when a team is not working from the same location. But is it the case in BI / DW scrum teams?
    In most cases one person is assigned to one task and they really don't care about listening to what other guys are doing.
  • According to the scrum principles each sprint should be finished with a demo for the business and a retrospective. It's great when a fancy report can be shown but in most cases it's the last part of the chain of sprints. In most BI implementation projects there might be tens of sprints when the backend is built and then at the very end the teams start to deliver results visible to the business owners.
    So might need to reconsider having a demo session when a developer shows the structure of oracle tables or a bunch of etl mappings or a chain of SQL statements to people for whom the biggest technical skill gained during their professional life is ability to copy an excel file from one folder to another.
  • The pressure to show business results quickly (it should be agile at the end right?) in many cases leads to focusing on development of those quick-win solutions which then leads to the creation of ad-hoc / temporary solutions which last for ages.
  • Micromanagement, status meeting, work time tracking, justification of a developer's working day. Scrum daily standup is not a status meeting! Scrum master is not a boss who's aim is to find out who was performing well at work yesterday and who had a less productive dya.